Generating your own certificate

* openssl req -newkey rsa:2048 -nodes -keyout myCompany.key -out myCompany.csr
* openssl x509 -signkey myCompany.key -in myCompany.csr -req -days 365 -out myCompany.crt

Get versions of python modules installed

 pip freeze | egrep 'pandas|pandas_market_calendars'

if you want to install a specific version:

pip install pandas_market_calendars==4.5.1

ssh banners

 This command will output SSH banners from the server as pseudo-terminal is not allocated.

#!/bin/bash

ssh <user>@<server> << EOF

echo 'hi A'

EOF

In order to remove the banner, we can either change the command as follows:

#!/bin/bash

ssh <user>@<server> 'exec bash' << EOF

echo 'hi A'

EOF

or easiest way is 

touch ~/.hushlogin

Zip file exploration

 in data exploration, when you have a lot of zip files, you need a way to look at the cotents with extracting each zip file.

• zcat file.log.zip | less
• unzip -p archive.zip file1.txt | less

OR the best way

• vim archive.zip

VSCode (Not Visual Studio)

 

• Shortcuts:
• Ctrl+P - Open a file
• Ctrl+Alt+F - Search for a file in Explorer

• Extensions:
• Remote Explorer - SSH
• 
  
• Git Lense - Git integration
• Data wrangler - Table view of data frames
• Zip Tools - Exploring zip files in a data set
• Windsurf / Codium - Auto Completion
• R-debugger via command line : install.packages("vscDebugger", repos = "https://manuelhentschel.r-universe.dev") # r-debugger vscode extension

Sending email from Python

 Can setup SMTP inside python itself but for better protection, setup sendmail with SMTP as in other post and then:

import subprocess

import os

import argparse

from email.mime.text import MIMEText

from email.mime.multipart import MIMEMultipart

from email.mime.base import MIMEBase

from email import encoders

def send_file_as_email(file_path, to, subject):

    """Sends an HTML file as the email body and also as an attachment."""

   

    # If subject is None, use the base file name

    if subject is None:

        subject = os.path.basename(file_path)

    # Create the email with mixed content (text + attachments)

    message = MIMEMultipart("mixed")

    message["Subject"] = subject

    message["To"] = to

    # Read HTML content and attach as email body

    with open(file_path, "r") as file:

        html_content = file.read().strip()

    html_body = MIMEText(html_content, "html")

    message.attach(html_body)

    # Attach the file as an attachment

    with open(file_path, "rb") as file:

        attachment = MIMEBase("application", "octet-stream")

        attachment.set_payload(file.read())

    # Encode to base64

    encoders.encode_base64(attachment)

    attachment.add_header("Content-Disposition", f'attachment; filename="{os.path.basename(file_path)}"')

    message.attach(attachment)

    # Send email using sendmail

    process = subprocess.Popen(["/usr/sbin/sendmail", "-t", "-i"], stdin=subprocess.PIPE)

    process.communicate(message.as_bytes())

    print("Email sent successfully!")

def main():

    """Parses command-line arguments and sends an email."""

   

    parser = argparse.ArgumentParser(description="Send an HTML file as an email body & attachment.")

    parser.add_argument("-f", "--file", required=True, help="Path to the HTML file")

    parser.add_argument("-t", "--to", default="harshal.patil@firl.com", help="Recipient email address (default: harshal.patil@firm.com)")

    parser.add_argument("-s", "--subject", default=None, help="Email subject (default: file name)")

    args = parser.parse_args()

    send_file_as_email(args.file, args.to, args.subject)

if __name__ == "__main__":

    main()

Bloomberg BQL

 BQL is not part of daily limit. So we can use those to get the data even when limit is hit:

=@BQL.QUERY("get(" & TEXTJOIN(",", TRUE, B1:E1) & ") for(members('RAY Index'))","cols=7;rows=5919")

where cells B1 to E1 have fields as follows:

market_cap px_volume 30_day_average_volume_at_time factor_expected_daily_volume

Python Selenium Webdriver sample program

 this is a sample program using selenium to click a button:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.chrome.service import Service

from webdriver_manager.chrome import ChromeDriverManager

import time

# Set up Chrome options

chrome_options = webdriver.ChromeOptions()

chrome_options.add_argument("--headless")  # Run in headless mode (no UI)

chrome_options.add_argument("--disable-gpu")  # Disable GPU acceleration for headless mode

chrome_options.add_argument("--no-sandbox")  # Disable sandboxing for headless mode

chrome_options.add_argument("--disable-dev-shm-usage")  # Fix shared memory issues

# Set up ChromeDriver using Service (Selenium 4)

service = Service(ChromeDriverManager().install())

# Initialize WebDriver (Chrome) with the path to your ChromeDriver

driver = webdriver.Chrome(service=service, options=chrome_options)

try:

    # Open the website

    driver.get("http://omsprodapp:9900/oms/f#ListedWrapper")

    

    # Wait for the page to load (you might need to adjust this)

    time.sleep(5)

    # Find the "Blotter All" button by its ID and click it

    blotter_button = driver.find_element(By.ID, "blotAllButt")

    blotter_button.click()

    # Wait for the action to complete (adjust the sleep if needed)

    time.sleep(2)

    print("Successfully clicked 'Blotter All'!")

finally:

    # Close the browser after the task is completed

    driver.quit()

logger.sh

 #!/bin/bash

source /home/$LOGNAME/.env

# Ensure at least two arguments are provided

if [ "$#" -lt 2 ]; then

    echo "Usage: $0 <command> <file_path> [additional_args...]"

    exit 1

fi

# Extract filename from the second argument

filename=$(basename "$2")

# Define log directory and log file path

log_dir="/stage/$LOGNAME/logs/$filename"

log_file="$log_dir/$(date +%d).log"

# Create log directory if it doesn't exist

mkdir -p "$log_dir"

# Redirect STDOUT and STDERR to the log file (no output on terminal)

exec >> "$log_file" 2>&1

echo "Executing "$@" at `date`"

# Execute the provided command

eval "$@"

Windows 11 Pro cannot grab window when maximized

 Win + R -> "control access.cpl"

Make the mouse easier to use

UnCheck "Prevent windows from being automatically arranged"

Loading env file in bash

 If you want to load .env file before running a command in bash, we can use following utility. (Can use the just the command as well.)

14:54:00 harshal@HP:~$ set -a; source ~/.env; set +a; CMD

or

14:55:42 harshal@HP:~$ cat ~/load_dot_env.sh

set -a; source ~/.env; set +a

14:55:47 harshal@HP:~$ source ~/load_dot_env.sh 

14:55:49 harshal@HP:~$ 

 

Citrix workspace on Ubuntu 2404

Typically, we start Citrix with this command:

/opt/Citrix/ICAClient/selfservice

But I was getting this error :

harsshal@HP:~$ /opt/Citrix/ICAClient/selfservice 

/opt/Citrix/ICAClient/selfservice: error while loading shared libraries: libwebkit2gtk-4.0.so.37: cannot open shared object file: No such file or directory

But followed these steps and then it worked:

sudo add-apt-repository -y "deb http://gb.archive.ubuntu.com/ubuntu jammy main"

sudo apt update

sudo apt install libwebkit2gtk-4.0-dev

To enable middle click, Change the configuration file  :

sudo vim /opt/Citrix/ICAClient/config/All_Regions.ini:

MouseSendsControlV=False 

AutoHotKey programs

volume.ahk

#Requires AutoHotkey v2.0

Volume_Down::SoundSetVolume "-10"

Volume_Up::SoundSetVolume "+10"

Volume_Mute::SoundSetMute "-1"

mouse.ahk

#Requires AutoHotkey v2.0

waitTime := 5 ; Number of minutes to wait

Loop {

    Sleep waitTime * 60 * 1000

    MouseMove 10, 0, 20, "R"

    MouseMove -10, 0, 20, "R"

}

blotter.ahk

#Requires AutoHotkey v2.0  ; Ensure the script runs with AutoHotkey v2

SetTimer RunCommand, 3600000  ; Set a timer to run every 1 hour (3600000 milliseconds)

RunCommand()

{

    currentTime := FormatTime("HH")  ; Corrected FormatTime call

    if ((currentTime >= 7 and currentTime <= 15) or currentTime = 17)  ; Check if it's between 7 AM - 3 PM or exactly 5 PM

    {

        Run("C:/Users/hpatil/AppData/Local/Microsoft/WindowsApps/python.exe c:/Users/hpatil/webdr.py")

    }

}

or

 #Requires AutoHotkey v2.0  ; Ensure the script runs with AutoHotkey v2

SetTimer RunCommand, 60000  ; Run every 60 seconds (1 minute)

RunCommand()

{

    run_times := ["0715", "0815", "1000", "1340", "1500", "1700"]  ; Define allowed times

    currentTime := FormatTime(A_Now, "HHmm")  ; Get current time in HHMM format

    if HasVal(run_times, currentTime)  ; Check if currentTime is in the array

    {

        Run("C:/Users/hpatil/AppData/Local/Microsoft/WindowsApps/python.exe c:/Users/hpatil/webdr.py")

    }

}

HasVal(haystack, needle)

{

    if !IsObject(haystack)  ; Ensure it's an array before looping

        throw Error("Bad haystack!")  ; Use `Error` instead of `Exception`

    for value in haystack

        if (value = needle)

            return true  ; Return true if found

    return false  ; Return false if not found

}

Sending email via command line

Requirements:

sudo apt install mailutils

sudo apt-get install ssmtp

cat /etc/ssmtp/ssmtp.conf

mailhub=email.firm.com:25 <smpt server>

Then there are multiple ways to it 

cat temp

To: harshal.patil@adyne.com 

Subject: Some Notifying Email 

MIME-Version: 1.0 

Content-Type: text/plain 

Body of your email goes here. Hello world

cat temp | sendmail -i -t

OR

 echo -e "To: harshal.patil@firm.com\nSubject: Subject\nContent-type: text/html" | cat - /stage/qspc/projects/sas/tools/sas_slippage_outliers.html | sendmail -i -t

OR

52 07 * * 1-5 echo "To: harshal.patil@firm.com, harshal.patil@firm.com \nSubject: Slippage Outliers $(date +\%F) \nContent-type: text/html\n\n" > /tmp/email_body.txt;cat /stage/qspc/projects/sas/tools/sas_slippage_outliers.html >> /tmp/email_body.txt;cat /tmp/email_body.txt | /usr/sbin/sendmail -i -t > /stage/qspc/logs/sendmail/$(date +\%d).log 2>&1

If wanted to attach as a file (when html contains graphs):

 (cat /stage/qspcadmin/logs/sendmail/intraday_liquidation.header ; uuencode /stage/qspcadmin/projects/sas/tools/intraday_liquidation.html intraday_liquidation.html) | /sbin/sendmail -i -t

where 

$ cat /stage/qspcadmin/logs/sendmail/intraday_liquidation.header
#To: harshal.patil@firm.com, a.b@firm.com
To: harshal.patil@firm.com
Subject: Intraday Strategy Liquidation

if you wanted an image embeded in the bodt:

(

echo "From: sender@example.com"

echo "To: recipient@example.com"

echo "Subject: Embedded Image"

echo "MIME-Version: 1.0"

echo 'Content-Type: multipart/related; boundary="boundary123"'

echo ""

echo "--boundary123"

echo "Content-Type: text/html; charset=UTF-8"

echo ""

echo '<html><body>'

echo '<p>Here is an embedded image:</p>'

echo '<img src="cid:myimage" width="600" height="400">'

echo '</body></html>'

echo ""

echo "--boundary123"

echo "Content-Type: image/jpeg"

echo 'Content-Transfer-Encoding: base64'

echo 'Content-Disposition: inline; filename="image.jpg"'

echo 'Content-ID: <myimage>'

base64 image.jpg

echo "--boundary123--"

) | sendmail -t

Python / pandas last_bday

import pandas as pd

from pandas.tseries.offsets import BDay

today = pd.to_datetime('today')

last_bday = today - BDay(1) 

Python groupby

 There are 2 ways to normal results:

• use agg function

grp_df = df.groupby(['BookLevel4','Symbol','Direction','OrderId'], as_index=False).agg({'TxTime':['min','max','count'], 'FillQuantity':['sum']})

grp_df.columns = [''.join(col).strip() for col in grp_df.columns.values]

• use apply function 
• In case you use more than 1 column
• Wont have to normalize multi-level columns

grp_df = df.groupby(['BookLevel4','Symbol','Direction','OrderId'], as_index=False).apply(

        lambda s: pd.Series({

            "TxTimemin": s["TxTime"].min(),

            "TxTimemax": s["TxTime"].max(),

            "TxTimecount": s["TxTime"].count(),

            "FillQuantitysum": s["FillQuantity"].sum(),

            "FillVWAP": (s['FillPrice'] * s['FillQuantity']).sum() / s['FillQuantity'].sum()

        })

    )

my .bcrc

 scale=6

my .screenrc

All possible variables are listed here:

https://www.gnu.org/software/screen/manual/html_node/String-Escapes.html#String-Escapes

 Here are the contents of my screenrc:

# Change default scrollback value for new windows

defscrollback 10000 # default: 100

# no welcome message

startup_message off

# advertise hardstatus support to $TERMCAP

termcapinfo  * '' 'hs:ts=\E_:fs=\E\\:ds=\E_\E\\'

termcapinfo xterm* ti@:te@

vbell "off"

bindkey "^[OR" prev

bindkey "^[OS" next

hardstatus alwayslastline

hardstatus string '%{= kG}[%{G}%H%? %1`%?%{g}][%= %{= kw}%-w%{+b yk} %n*%t%?(%u)%? %{-}%+w %=%{g}][%{B}%m/%d %{W}%C%A%{g}]'

screen -t notebook 

screen -t local