We lost ~25 webhook events in 7 minutes because our Cloud Run service was scaling to zero. Here is how I used Cloud Scheduler to toggle min-instances during business hours — and the workaround for Cloud Scheduler not supporting PATCH requests.
We had a managed connector (think Fivetran, Airbyte, or similar) syncing our helpdesk data — tickets, contacts, threads, comments — from a SaaS helpdesk platform into BigQuery. It worked fine until it did not: the connector would miss webhook events, lag behind on incremental syncs, and occasionally produce phantom duplicates that our dbt models would faithfully propagate into dashboards. The cost was also non-trivial for what amounted to six tables.
We had four service accounts for BigQuery, one per GCP project. Each had its own JSON key file, its own set of IAM roles, and its own set of problems. When a local CLI tool needed to query production data, it used one key. When it needed to write to the raw data project, it used a different key. Our Slack bot had two keys mounted as secrets. The monitoring scripts had yet another. Every new tool or integration meant figuring out which key to use, and getting it wrong meant cryptic PERMISSION_DENIED errors that could take 20 minutes to debug.
Our BI tool does not expose a usage API. So I built a headless Playwright scraper on Cloud Run that logs into the BI platform, scrapes dashboard usage metrics, loads them into BigQuery, and sends weekly Slack summaries. Here is how I handled the hardest part: session management.
DRY Revenue Logic: Extracting a Shared dbt Macro for Partner-Specific Calculations
by Marc
written by Marc
Revenue numbers that don’t add up are the kind of bug that erodes trust fast. Last month I tracked down a case where a quote adjustment of over two thousand euros was producing zero revenue in our reporting layer. The root cause? Duplicated business logic across two dbt models — logic that had drifted apart over time.
When Your Application Mutates created_at: Fixing Timestamps with CDC History
by Marc
written by Marc
The application was silently rewriting created_at after record creation, making child records appear older than their parents. Here is how I caught it and fixed it using raw CDC history in the dbt staging layer.
I built a paper trading bot that runs on a Raspberry Pi 5, executes 8 strategies across stocks and crypto, and has been trading a simulated 10,000 euro portfolio since early 2026. This post covers the technical details: the strategy code, the backtester that validated the strategies against 195,000 candles, the yfinance pitfalls that cost me days of debugging, and the web dashboard that lets me monitor everything from my phone.
I spent two days hunting a revenue bug that produced numbers exactly 100 euros off on specific records. The SQL was correct. The data model was correct. The individual fields were correct. The bug was in how I was combining them — a subtle interaction between revision-aware bid fields and delta-based quote adjustment fields in a Unified Star Schema metrics bridge.
Our data team of four was drowning in ad-hoc requests. “What was the revenue last month?” “How many active customers do we have in Germany?” “What’s the conversion rate for Q4?” Each question meant someone opening a SQL editor, writing a query, formatting the results, and posting them in Slack. So I built an AI-powered data assistant: a Slack bot backed by Claude and BigQuery that lets anyone on the team ask business questions in natural language and get SQL-backed answers in seconds.
How two duplicate webhook events, arriving 2ms apart, broke 12 dbt tests in production — and the surprisingly simple fix that made the sync idempotent.