https://colab.research.google.com/drive/1lLpUyA6LdKLLXiptEDFLoIWEpek8xu5J?usp=sharing

(Because this is what we do now with AI).

(and i needed it for a deck).

bolt.new is great and i felt like a wizard at first just freestyling away on the keys but man what a mess i ended up creating - ended up down various dead ends with firebase and deployment to netlify and lots of infra related mess that ended up waaay too complex for me at that stage down the rabbirt hole to know where to start to try untangle anything.

so i jumped over to chatgpt o1 and just have an actual conversation and some back and forth - chatgpt was much more nuanced and less eager to just start cranking out code for me.

we took it step by step and i was actually learning some stuff at each stage rather than blind and wild js flying all over the place.

so ended up with an actual repo in gh thats deployed and a fairly decent starting point for what i want: https://github.com/andrewm4894/andys-daily-factoids

https://andys-daily-factoids.com

Anyway funny part is that all my factoids are about how Octopuses have 3 hearts! It really really loves this fact.

Also honey does not spoil is a strong 2nd favorite.

Next step is to add last 100 or so factoids in the prompt and ask it not to repeat itself…

You can listen to it here or just search “anomstack data eng” in wherever you listen to podcasts.

So below is one little recipie to do this, feel free to use and adapt however suits your needs.

This is required listening for anyone working with data - really great sort of inside baseball stuff too that you get via tidbits here and there as you listen that really all compounds over time.

And also Demetrios the host is glorious!

I always have it at the top of podcasts and YouTube channels I recommend to anyone in data from students starting off to old timers like myself (I’m embracing it).

Also the site and platform is really cool too: https://mlops.community/

p.s. i write this post after about an hour trying to find one single decent platform (except apple obviously) to leave a review to try show support for the show. In the end it was too painful so I wrote this ¯\_(ツ)_/¯

So I decided to try get going with it in as minimal a way as possible using everyone’s favourite dataset.

https://github.com/andrewm4894/learn-malloy

To get going all you need to do is:

• install malloy vscode extension

• that’s pretty much it, each .malloy or .malloynb file will make it obvious how to run queries or run usual notebook style flow.

Moving parts

• a source object defining some sort of data connection (in this case just a local csv) and, optionally, some additional measures and dimensions.

• some queries against a source. This is where i think it essentially becomes a sort of DSL (and more) on top of SQL.

• optionally some style objects that can help map queries to output visualisations. This is what’s behind the magic of some of the default visualizations in the cool demos here.

• even the ability to bring it all together into a familiar notebook experience.

Initial thoughts

• Seems like a really interesting project and idea (no doubt a lot has been learned from look.ml and those learnings will be baked into this project).

• Seems to have a lot of the abstractions right imo - feels natural in some sense while also still quite flexible.

• I was able to just guess about 70% of what I wanted or expected when writing some example queries.

• Can’t wait until I can use this natively in something like BigQuery UI and easily share results of analysis with co-workers.

• Seem to be a cool use case for duckdb in my csv example - need to learn more how that’s all working.

• I could see this becoming a nice standard way to package up and share analysis type projects with bosses and colleagues in a way that they can easily then play with, customize and build on if they have follow on questions - so i no longer become the bottleneck - great!

• Could also see this be quite a useful tool to translate and define business questions with stakeholders without having to show them any sql!

• One of my hot takes over last few years was that SQL is the new excel, but maybe Malloy could also be the new excel.

You don’t need to run off and buy an (expensive!) subscription for the latest hot data observability Sass offering (there is lots and some of them are great in my experience, Metaplane does a lot really well here in my opinion but i am of course not familiar with all of tools - obligatory crazy landscape picture).

If instead, you want to keep things simple to begin with, you probably already have most of the main ingredients you need for a pretty decent anomaly detection stack on whatever metrics you want - using our old friend Airflow and this new anomaly detection provider package I built because I just love anomaly detection that much.

(I really do - metrics are everywhere so we need decent anomaly detection to avoid overload, anomaly detection is a great mix of art and science as can be quite subjective, and decent anomaly detection should some-day kill all dashboards!!!)

Anomaly detection Airflow provider

So here is the TL;DR; (if curious for more detail check out the project README on GitHub).

An Airflow Provider for Anomaly Detection:

1. You define “metrics batches” in some sql (for example, here is the “metrics_hourly” batch in the example dag).

2. Some yaml configuration fun (you can just use the defaults) for params of config you might want to change (for example, here is the config for the “metrics_hourly” batch).

3. You get some alerts via email (which you can just use for Slack etc too) when metrics look anomalous.

How it works

Really all this provider and example dag are doing is creating a set of 4 dags within airflow, one set for each “metric batch” you define (different “metric batches” can be for different frequency or subject areas - you can use them however you want as each metric batch is basically just templated jinja sql files and a corresponding yaml config file):

1. Ingestion: Ingest each metric batch by running the sql and just appending the results (the metrics) to some table you can pointed it at.

2. Training: Train an anomaly detection model, one per metric within each metric batch, and save the trained model to a Google Cloud Storage (GCS) bucket.

3. Scoring: Use the model trained in step 2 to score recent data and save scored metrics to another table you have pointed it at.

4. Alerting: A dag that looks over recent scored metrics and just alerts based on traditional enough rolling thresholds on those scored metrics.

Pros vs. Cons

Here are some pro’s and con’s to help you decide if using and getting involved in this project makes sense to you.

Pros:

• You bring your SQL to define your metrics and let this provider do the rest.

• Nice and simple alerts.

• A lot of us are already using Airflow for ETL type stuff so we are quite comfortable with it.

• This just packages up the problem into a few Airflow dags and so makes it all a bit easier to reason about and understand (as well as customize if you like).

• A lot of us also have business metrics in cloud data stores like Google BigQuery, AWS Redshift, and Snowflake etc so we don’t actually need to involve a new database layer in any of this - just let Airflow do the orchestration and work for ingestion, training and scoring.

Cons:

• Better suited for metric batches of stuff that does not need to be near real time - eg. hourly or daily business metrics are very well suited (it’s what I built this for).

• Still a fairly immature project so more a case of try it yourself and see how it goes (I’d love some help :) ).

• Some limits to the size and complexity of models that make sense - from my experience decent enough anomaly detection is totally possible with reasonable traditional ML models and sensible feature engineering.

• Don’t really have the ability to build a workflow into the anomaly alerts just yet - you get an email when something looks strange and that’s that - we don’t have any incident management type workflows as part of this since that would be another fairly complex moving part we are avoiding for now.

Get involved!

This project is very very easy to get involved in and I hope could end up being quite useful for the wider Airflow community - it’s crazy that we don’t really have easy ways to just get some decent anomaly detection on all our business metrics within the same context of the ETL’s and dags we create all the time in Airflow.

The basic core idea is, le’ts just use Airflow for a bit more than ETL’s - one way to frame the anomaly detection use case is just some ETL’s and a few small python tasks so Airflow is a perfect tool for this.

Feel free to make issues and PR’s in the Github repo: https://github.com/andrewm4894/airflow-provider-anomaly-detection

Useful links

• Project README

• Example alert - An example of what an alert looks like.

• Anomaly gallery - Some real world example i have been adding while dogfooding.

• Example dag - The example dag that ships with the provider. Best place to start and look around if just want to see what’s going on and already familiar with Airflow etc.

• Getting started - How to play around and try it out yourself.

Here is a minimal enough example repo using Terraform and GCP Functions Framework to build a GCP Python function that will receive a Stripe webhook event, perform signature verification, and then just print the event. You can adapt and build whatever logic you want then on top of this.

TL DR; if you are looking at the stripe docs and cant figure out why you can run your python function locally but the signature verification fails once deployed to GCP functions (with this generic error message error.SignatureVerificationError( stripe.error.SignatureVerificationError: No signatures found matching the expected signature for payload) it could be that you need to replace payload = request.data with payload = request.data.decode('utf-8'). This i found out after a day or two thanks to this SO comment.

Most of what might be useful is in the repo readme, but below i’ll quickly walk through the structure and moving parts.

Repo structure

Here are the main folders and files involved. Greyed out files are those that might have sensitive info and so are part of the `.gitignore` and so not to be committed to source control. I’ll quickly walkthrough each folder and important files below.

• /python-functions - The python code for the function lives in here.
  • /stripe_webhook - A folder just for the “stripe_webhook” function.

    • main.py - Python source code for the function.

    • requirements.txt - Python libraries the function needs to run.

  • /zipped - A local folder of zipped up versions of the python function folders - this zip is what will be loaded to a GCS bucket as the source for the cloud function. This is ignored from source control and not needed.
• /terraform - All terraform related code and configuration files live in here to provision the cloud function and related cloud resources.

  • conf.tf - A file defining some confidential vars you want available to terraform but not in source control. See conf.example.tf for dummy example and instructions.

  • gcp-cloud-functions.tf - Terraform code to provision, configure and deploy all related cloud resources used by the function.

  • gcp-secret-manager.tf - Terraform code to define a GCP secret used by the cloud function. The Stripe secret will live in GCP secret manager.

  • terraform.tf - Some standard boilerplate used as part of Terraform set up and when initializing with `terraform init`

  • variables.tf - Some non-sensitive variables we want to use in defining our terraform resources.

• /venv - Our local python development virtual environment used for testing and debugging the function locally using the functions framework cli.

• .env - A env file that will be picked up when running the function locally so that the stripe_endpoint_secret environment variable will be available locally (coming from .env file) and when running in GCP cloud (from GCP secrets manager). Also make sure if not under source control, see .example.env for a dummy example.

• .gitignore - Things we want to make sure we don’t add to source control.

• requirements.txt - Python libraries we want to install into our `venv` for local execution and debugging of the function.

Run function locally

To run the function locally we can use the functions framework cli like below:

# run function locally in debug mode on port 8081
functions-framework --source=./python-functions/stripe_webhook/main.py \
  --target=stripe_webhook \
  --debug \
  --port=8081

This should return something like below to show the function running locally on port 8081 (you can use whatever port you want).

(venv) PS > functions-framework --source=./python-functions/stripe_webhook/main.py \
  --target=stripe_webhook \
  --debug \
  --port=8081
 * Serving Flask app 'stripe_webhook'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8081
Press CTRL+C to quit
 * Restarting with watchdog (windowsapi)
 * Debugger is active!
 * Debugger PIN: XXX-XXX

Stripe CLI

Set up local forwarding

Once the function is running locally you can use the stripe cli to (1) forward events to a local endpoint and then (2) trigger some test events to see the response.

# once function is running locally you can forward events to local endpoint
stripe listen --forward-to localhost:8081

You should see something like this when local forwarding is set up:

PS C:\Users\andre> stripe listen --forward-to localhost:8081
> Ready! You are using Stripe API Version [2022-11-15]. Your webhook signing secret is xxx_xxxxxx (^C to quit)

Create some test events

# create a test event
stripe trigger payment_intent.succeeded

You should see something like this for a successful test event creation:

PS C:\Users\andre> stripe trigger payment_intent.succeeded
Setting up fixture for: payment_intent
Running fixture for: payment_intent
Trigger succeeded! Check dashboard for event details.

If the function as been invoked successfully then in the window from step 1 above you should see something like this:

PS C:\Users\andre> stripe listen --forward-to localhost:8081
> Ready! You are using Stripe API Version [2022-11-15]. Your webhook signing secret is xxx_xxxxxx (^C to quit)
2022-12-22 12:31:10   --> charge.succeeded [evt_3MHnvQFE3Qfj39xW1UE7UhT6]
2022-12-22 12:31:10   --> payment_intent.succeeded [evt_3MHnvQFE3Qfj39xW1vzt9gAn]
2022-12-22 12:31:10   --> payment_intent.created [evt_3MHnvQFE3Qfj39xW1KR01wQ8]
2022-12-22 12:31:10  <-- [200] POST http://localhost:8081 [evt_3MHnvQFE3Qfj39xW1UE7UhT6]
2022-12-22 12:31:11  <-- [200] POST http://localhost:8081 [evt_3MHnvQFE3Qfj39xW1vzt9gAn]
2022-12-22 12:31:11  <-- [200] POST http://localhost:8081 [evt_3MHnvQFE3Qfj39xW1KR01wQ8]

Finally in the window where you triggered the functions framework to run you should just see the a json string with all the event info itself.

(venv) PS C:\Users\andre\Documents\repos\stripe-webhook-gcp-function> functions-framework --source=./python-functions/stripe_webhook/main.py --target=stripe_webhook --debug --port=8081
 * Serving Flask app 'stripe_webhook'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8081

Press CTRL+C to quit
 * Restarting with watchdog (windowsapi)
 * Debugger is active!
 * Debugger PIN: 107-679-323
{"id": "evt_3MHnvQFE3Qfj39xW1UE7UhT6", "object": "event", "api_version": "2022-11-15", "created": 1671712265, "data": {"object": {"id": "ch_3MHnvQFE3Qfj39xW1ljeXQ0U", "object": "charge", "amount": 2000, "amount_captured": 2000, "amount_refunded": 0, 
...
"name": "Jenny Rosen", "phone": null, "tracking_number": null}, "source": null, "statement_descriptor": null, "statement_descriptor_suffix": null, "status": "requires_payment_method", "transfer_data": null, "transfer_group": null}}, "livemode": false, "pending_webhooks": 4, "request": {"id": "req_BsagVXD6LQwqjH", "idempotency_key": "e36a0855-a9e6-441a-b9ca-181632fd43ad"}, "type": "payment_intent.created"}
127.0.0.1 - - [22/Dec/2022 12:31:11] "POST / HTTP/1.1" 200 -

Thats it

That’s it, you can then add whatever additional logic you want to handle specific stripe webhook events and control what events make it to this webhook from within the stripe ui itself as needed.

The developer tools and experience with Stripe is really good and being able to so easily run a cloud function locally with realistic test data from stripe using these two cli’s (stripe and functions-framework) is really nice, even if it did take me a few days to realize I needed to use that .decode('utf-8') once the function was deployed to GCP cloud - there’s always going to be something that trips you up a little :)