I have recently needed to watch and track various activities on specific github repos i'm working on, however the rest api from Gtihub can sometimes be a bit limited (for example, best i could see, if you want to get the most recent list of people who began watching your repo you need to make … Continue reading Github Webhook -> Cloud Function -> BigQuery
Papers i’m reading #1
I've recently set myself the goal of reading one academic paper a week relating to the ML/AI things i'm working on i'm my current role. To try help keep me honest and diligent in this regard, I've decided to get into the habit of jotting down some quick notes on each paper and every now … Continue reading Papers i’m reading #1
My First PyPI Package
I've been threatening to myself to do this for a long time and recently got around to it, so as usual i'm going to try milk it for a blog post (Note: i'm not talking about getting into a box like the below picture, its something much less impressive). Confession - I don't know matplotlib … Continue reading My First PyPI Package
KubeFlow Custom Jupyter Image (+ github for notebook source control)
I've been playing around a bit with KubeFlow a bit lately and found that a lot of the tutorials and examples of Jupyter notebooks on KubeFlow do a lot of the pip install and other sort of setup and config stuff in the notebook itself which feels icky. But, in reality, if you were working … Continue reading KubeFlow Custom Jupyter Image (+ github for notebook source control)
Multi-Variate, Multi-Step, LSTM for Anomaly Detection
This post will walk through a synthetic example illustrating one way to use a multi-variate, multi-step LSTM for anomaly detection. Imagine you have a matrix of k time series data coming at you at regular intervals and you look at the last n observations for each metric. A matrix of 5 metrics from period t to t-n One approach … Continue reading Multi-Variate, Multi-Step, LSTM for Anomaly Detection
Custom Python Packages in AWS Lambda
It's True. I'm pretty sure i'll be looking this up again at some stage so that passed one of my main thresholds for a blog post. I've recently been porting some data and model development pipelines over to AWS Lambda and was mildly horrified to see how clunky the whole process for adding custom python … Continue reading Custom Python Packages in AWS Lambda
Clustering Cell Tower Usage Data
Below is a little mini project i did a while back looking at cell tower usage data. Main takeaway really was a nice example of how subjective clustering can be, especially the more features and variables you feed into your distance metric.
Parallelize a wide df in Pandas
I was going to make a pretty picture. Sometimes you end up with a very wide pandas dataframe and you are interested in doing the same types of operations (data processing, building a model etc.) but focused on subsets of the columns. For example if we had a wide df with different time series kpi's … Continue reading Parallelize a wide df in Pandas
( 0 – 0 ) / 0 != 0
Arrrgghh - I just wasted the best part of an afternoon chasing this one down. If i can knock out a quick post on it then at least i'll feel i've gotten something out of it. Here's the story - somewhere in an admittedly crazy ETL type pipeline i was using pandas pct_change() as a … Continue reading ( 0 – 0 ) / 0 != 0
Java Hello World cli using args4j
A simple example of how to use args4j to add command line args to a simple "Hello World" type java application. For example below command line execution would print "Hello arg4j!" instead of the default "Hello World!" if you don't pass any args. $ java -jar helloWorldParamaterized --msg='Hello arg4j!' https://gist.github.com/andrewm4894/0e9d741126d7c8add261724868d4a844