Given a text file we want to create another file containing only those lines that match a certain regular expression using python3 1 2 3 4 5 6 import re with open("./in.txt", "r") as input_file, open("out.txt", "w") as output_file: for line in input_file: if re.match("(.*)import(.*)", line): print(line, file=output_file)
Some useful resources about Airflow: ETL best practices with Airflow Series of articles about Airflow in production: Part 1 - about usecases and alternatives Part 2 - about alternatives (Luigi and Paitball) Part 3 - key concepts Part 4 - deployment, issues More notes about production About start_time: Why isn’t my task getting scheduled? One cannot specify datetime.now() as start_date. Instead one has to provide datetime object without timezone. Probably UTC time is required....
Conversion from calendar week to date Sometimes one has to convert a date written as year, calendar week (CW), and day of week to an actual date with month and date. The behaviour in the begin/end of a year may be not straightforward. For example according to ISO 8601 monday date of the CW 1 year 2019 is 31 January 2018. As far as I can see there is no standard function for conversion in python....
Awesome explanation of SQLAlchemy with examples and comparison to Django by Armin Ronacher: SQLAlchemy and You Flask-SQLAlchemy module Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application. How to add SQLAlchemy to Flask application: from flask import Flask from flask_sqlalchemy import SQLAlchemy app = Flask(__name__) # configuration of the DB is read from flask configuration storage app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db' # here we define db object that keeps track of sql interactions db = SQLAlchemy(app) Now we are ready to define tables and objects using predefined db....
Pytest is a powerful tool for testing in python. Here are some notes about hands-on experience. Running tests in pytest with/without a specified mark Execute pytest -m "integration" if you want to run only tests that have “@pytest.mark.integration” annotation. Similarly you can run only tests that don’t are not marked. pytest -m "not your_mark" That command will test everything that is not marked as “your_mark”. How to verify exception message using pytest One can use context manager pytest....
I want to use EmberJS with Flask application. Flask will provide an API. Ember frontend will consume and display data from the Flask backend. Let’s say we want our fronted to display a list of users of the system. We will have a main page and users page in our frontend. On the users page the client will see a list of users that we get from backend. Source code is here: ember_flask_example...
Build time can be reduced by using multiple cores of your processor. Parallelization for make: make -j8 Parallelization for cmake: cmake --build <bindir> -- -j 8 Sometimes you cannot pass parameters directly to make. For example you are installing a python module using python setup.py install Setup script doesn’t accept parameters. Then you could pass them through environment variables: export MAKEFLAGS="-j 8" see also intro to CMake video, russian.
1 2 3 4 5 import datetime from dateutil.relativedelta import relativedelta your_datetime = datetime.datetime.now() your_datetime + relativedelta(months=1) # adds one month That function clips the overflowing day of the months: 1 2 3 4 5 6 7 8 9 10 >>> datetime(2021, 1, 28) + relativedelta(months=1) datetime.datetime(2021, 2, 28, 0, 0) >>> datetime(2021, 1, 29) + relativedelta(months=1) datetime.datetime(2021, 2, 28, 0, 0) >>> datetime(2021, 1, 30) + relativedelta(months=1) datetime.datetime(2021, 2, 28, 0, 0) >>> datetime(2021, 1, 31) + relativedelta(months=1) datetime....
Import module by path/name 1 2 import importlib module = importlib.import_module(module_path) Find class implementing certain interface in a module 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 def load_class_from_module(module_path: str, target_class: type) -> type: module = importlib.import_module(module_path) list_classes = get_all_classes_implementing_interface(module, target_class) if not list_classes: raise RuntimeError("Unable to find implemented interface {} in {}".format(target_class, module_path)) if len(list_classes) > 1: raise RuntimeError("More then 1 implementation of {} was found in {}"....
How to include default values in ‘–help’ 1 2 3 parser = argparse.ArgumentParser( # ... other options ... formatter_class=argparse.ArgumentDefaultsHelpFormatter) Output looks like this: 1 2 3 --scan-time [SCAN_TIME] Wait SCAN-TIME seconds between status checks. (default: 5) Another tip: add ‘%(default)’ to the help parameter to control what is displayed. 1 2 parser.add_argument("--type", default="toto", choices=["toto","titi"], help = "type (default: %(default)s)") source