Guruh

January 03, 2021

Python Cheat Sheet

Python snippets I reach for often when working with data.

I recently worked on GitLab runner resource budgeting. The team had a large volume of container CPU and memory telemetry to review. My job was to find under utilized container resources and adjust them accordingly. So I had to pull out insights from those data and have multiple discussions with my team.

I knew it would be all about analysis, I immediately pulled up my jupyter notebook, despite I don’t use python regularly. I started listing things that I needed to do. Requirements were lining up, I knew what to do, this is going to be the best of the night, I started right away.

But then things began to rattle a little bit when I forgot to do particular things in python. I forget how to subtract 2 datetimes, I don’t remember how to read a file in python. I have to open my previous works which spill out in many different places just to remember these basic things. I grew tired of rediscovering the same snippets every time I needed them.

Here are the Python patterns I use most often.


Files

Read a file

with open("file.txt") as f:
   data=f.read()

print(data)

Write to a file

foo="bar"
with open("foo.txt", "w", encoding="utf-8") as f:
    f.write(foo)

Read a json file

import json
###
# file.json sample
# [{foo: "bar"}, {foo: "bar"}]
###

with open("file.json", "r") as f:
  file_json=json.loads(f)

for d in file_json:
  print(d["foo"])

Datetime

Substract two dates

# initializing a datime
to_date = datetime(2020, 12, 17, 23, 59)

# substract to_date by 10 days
from_date = (to_date - timedelta(days=10))

Convert to Javascript Date (JSON Date)

When we want to convert a python datetime to a javascript date, we have to times the total seconds by 1000. As strftime method formats a date into a string object, one of the ways to have a correct json date is to append it with tripple zeros. Or you can cast it into an integer then you multiply it by 1000.

json_date = to_date.strftime("%s") + "000"

json_date = int(to_date.strftime("%s")) * 1000

Pandas

If you’re currently working with spreadsheet or excel, I highly recommend you to start taking a look at pandas. Pandas is like a spreadsheet with powerful python data analysis tools upon it.

Pandas adoption is so fast. The veil between your ideas and the working code is so remarkably thin. This is the main reason why it is so convenient to use pandas whenever I need to do something with data.

These are magical pandas tricks that I use quite a lot and I love them so much.

Functions

Apply a function to a column

def func(param):
  return param

df.col.apply(lambda d: func(d))
df.col.apply(func)

Apply a function to multiple columns

def func(df):
  return df["a"] + df["b"]

df.apply(func, axis=1)

Sort values

It’s surprising that we can sort a dataframe by multiple columns.

df = df.sort_values('name')

df.sort_values(by=["col1", "col2"], ascending=False)

docs:

Count values

If you need a way to group a list with their total occurrences, this is one way to do it.

df.col.value_counts()

df.col.value_counts(dropna=True)

docs:

Join or Merge dataframes

Pandas often replaces ad hoc SQL for exploratory work. SQL queries can become quite intrusive when you keep adding more filters and calculations into them. Sometimes, I think to myself SQL queries are only fitted for applications needs.

df_left = pd.DataFrame()
df_right = pd.DataFrame()

how = "left" | "right" | "inner" | "outer"
merged_df = pd.merge(df_left, df_right, left_on=["key"], right_on=["Key"], how=how)

docs:

Drop columns

df.drop(["col", "col-1"], axis=1, inplace=True)

Dealing with na / null values

import pandas as pd

df.fillna('', inplace=True)

df.dropna(inplace=True)

df.isna()
df.col.isna()

docs :


Numpy

Split an array to multiple arrays (Chunks)

data=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunks = np.array_split(data, 3)
print(chunks)
# [array([1, 2, 3, 4]), array([5, 6, 7]), array([ 8, 9, 10])]

That’s all folks, I hope you find them useful. More snippets might be added to the list in the future.

Thanks for reading. See you in the next post.