Libraries Standard multiprocessing Pebble - pretty close to the standard one, but with a bit nicer interface Dask - well maintained and (almost) drop-in replacement of numpy and pandas: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Arrays implement the Numpy API import dask.array as da x = da.random.random(size=(10000, 10000), chunks=(1000, 1000)) x + x.T - x.mean(axis=0) # Dataframes implement the Pandas API import dask....
command line arguments is a standard and one of the most common ways to pass parameters to a python script. There exist a list of python libraries that help with that task. Here I am going to list some of them. argparse --------------------------- The default choice for the python developer. The module is included in python standard library and comes together with any python distribution. Example of usage: .. code-block:: #!...
Basics Connect as user postgres: 1 psql -U postgres Connect to a specific database: \c database_name; Quit the psql: \q List all databases: \l Lists all tables in the current database: \dt List all users: \du Create a new role username with a password: 1 CREATE ROLE username NOINHERIT LOGIN PASSWORD password; Managing tables Create a new table or a temporary table 1 2 3 4 5 6 7 CREATE [TEMP] TABLE [IF NOT EXISTS] table_name( pk SERIAL PRIMARY KEY, c1 type(size) NOT NULL, c2 type(size) NULL, ....
Given a text file we want to create another file containing only those lines that match a certain regular expression using python3 1 2 3 4 5 6 import re with open("./in.txt", "r") as input_file, open("out.txt", "w") as output_file: for line in input_file: if re.match("(.*)import(.*)", line): print(line, file=output_file)
Some settings I find useful for a workstation CPU monitoring on the main panel Default Ubuntu desktop seems to become finally convenient enough for me starting from Ubuntu 18.04. Only several tweaks are missing. Constantly available CPU/Mem/HDD/Network monitor is one of them. Here is how to install a small widget for a top panel in the default GNOME desktop environment. sudo apt-get install gir1.2-gtop-2.0 gir1.2-networkmanager-1.0 gir1.2-clutter-1.0 Go to Ubuntu Software and then search for system monitor extension....
Pytest is a powerful tool for testing in python. Here are some notes about hands-on experience. Running tests in pytest with/without a specified mark Execute pytest -m "integration" if you want to run only tests that have “@pytest.mark.integration” annotation. Similarly you can run only tests that don’t are not marked. pytest -m "not your_mark" That command will test everything that is not marked as “your_mark”. How to verify exception message using pytest One can use context manager pytest....
Some linux commands that I’ll probably need in the future User management Add user to a group sudo usermod -aG group user or sudo adduser <user> <group> Delete user userdel user Delete the user’s home directory and mail spool: userdel -r user Remove user from a group sudo gpasswd -d user group list all users: $ getent passwd list all groups: $ getent group list all groups of the current user:...
Using convert utility we can join multiple image files (png, jpeg, pdf) into one pdf. convert -density 250 file1.pdf file2.JPG file3.pdf -compress JPEG -quality 70 combined.pdf we use JPEG compression with quality =90 for images inside PDF. Otherwise the PDF will be too large. source For me images appear rotated inside the pdf. That’s probably because EXIF information about image rotation is not taken into account by pdf converter. You can remove all the meta information first with convert <input file> -strip <output file> and then combine it to a pdf....
Create more secure ssh keys Create a key using elliptic curve cryptography (more secure): ssh-keygen -a 100 -t ed25519 generate RSA key of length 4096 to file my_key ssh-keygen -t rsa -b 4096 -C "your@e-mail.com" -f my_key Generate md5 fingerprint of the key (works in newer ubuntu, 16): ssh-keygen -lf ./my_key -E md5 see also: Upgrade Your SSH Key to Ed25519 How to prevent SSH from scanning all the keys in ....
Super harsh guide to machine Learning Super harsh guide to machine learning (reddit) First, read fucking Hastie, Tibshirani, and whoever. Chapters 1-4 and 7. If you don’t understand it, keep reading it until you do. You can read the rest of the book if you want. You probably should, but I’ll assume you know all of it. Take Andrew Ng’s Coursera. Do all the exercises in Matlab and python and R....