Image and video processing recipes

I often need to perform some operations on video. Usually I use linux and ffmpeg to do that, but I strugle to remember all the commands. Therefore I usually google it. I'll try to collect here some recipes that I usually use with a bit of explanations.

Video conversions

Cut …

more ...

C++ and CMake

CMake in VSCode

VSCode is a free open source IDE with a lot of nice features. In addition one can chose from a variety of extensions. I couldnt find a good extension for CMake integration that works out of box. I tried to make use of Cmake-tools and it kind …

more ...



Capture PWM signal using Arduino

Parsing PWM signals

For my robocar project I needed to understand the mechanism of pulse width modulation of the remote control.

My intention was to use Arduino as a proxy between RC-receiver and servos/ESC to be able to record the used input for imitation learning. Human driver (me) sends …

more ...


Robocar project

For the robocar contest in Berlin I started a project of building an autonomous toy car (scale 1:10). The goal of the contest was to show the fastest lap driving autonomously. The track had 8-shape with lane boundaries marked with white tape.

Unfortunately the competition got cancelled. But that …

more ...


Lenovo E490 memory upgrade

Lenovo e490 is a cheaper and less performant alternative to the famous T series.

The laptop came to me with a deformed box but I decided to give it a try and so far so good.

lenovo e490

I have bought a basic version and upgraded RAM for it. Here are some …

more ...


Computer vision libraries

So, what else is there except for opencv...

CCV

CCV website, github

CCV 0.7 comes with a sub-10% image classifier, a decent face detector.

It runs on Mac OSX, Linux, FreeBSD, Windows*, iPhone, iPad, Android, Raspberry Pi. In fact, anything that has a proper C compiler probably can run …

more ...

Postgres cheatsheet

Basics

Connect as user postgres:

psql -U postgres

Connect to a specific database:

\c database_name;

Quit the psql:

\q

List all databases:

\l

Lists all tables in the current database:

\dt

List all users:

\du

Create a new role username with a password:

CREATE ROLE username NOINHERIT LOGIN PASSWORD password …
more ...

Jupyter notebooks on EMR

Explanatory data analysis requires interactive code execution. In case of spark and emr it is very convenient to run the code from jupyter notebooks on a remote cluster. EMR allows installing jupyter on the spark master. In order to do that configure "Applications" field for the emr cluster to contain …

more ...

Spark on a local machine

How to install spark locally

Considering spark without hadoop built-in.

  • Download hadoop unpack to /opt/hadoop/
  • Download spark without hadoop, unpack to /opt/spark
  • Install java. Set JAVA_HOVE environment variable. For example: export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  • create environment variables required for spark to run. One can put those …
more ...


Reducing disk usage in Ubuntu

Here are some recipies to make ubuntu installed on USB drive to work faster.

[1]

[2]

reducing swapping

Add these lines to /etc/sysctl.conf, and reboot.

vm.swappiness = 0
vm.dirty_background_ratio = 20
vm.dirty_expire_centisecs = 0
vm.dirty_ratio = 80
vm.dirty_writeback_centisecs = 0

More caching while writting on disk

Add noatime …

more ...

Spark in Docker with AWS credentials

Running spark in docker container

Setting up spark is tricky. Therefore it is useful to try out things locally before deploying to the cluster.

Docker is of a good help here. There is a great docker image to play with spark locally. gettyimages/docker-spark

Examples

Running SparkPi sample program (one …

more ...

OKRs

We had a production system written by mathematicians, 50 different stakeholders with conflicting targets, five leadership changes during last year, a dozen of microservices, AWS costs of 10 thousands per week,
hole galaxy of legacy databases, cron jobs, Celery, greenlets, … Also, unstable API as a dependency, 10 Gb of text …

more ...

Bokeh in jupyter notebooks for interactive plots

Bokeh is a library for interactive visualization. One can use it in Jupyter notebooks.

Here is the example.

Lets say we have a pandas dataframe with timestamps and some values:

import pandas as pd
from io import StringIO
df = pd.read_csv(StringIO("""timestamp,value
2018-01-01T10:00:00,20
2018-01-01T12:00 …
more ...

Comparison of click-based config parsers for python

Problem

There is click module that allows you to create comman dline interfaces for your python scripts. The advantages of click are

  • nice syntax
import click

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
              help='The person to greet.')
def hello …
more ...