We had four service accounts for BigQuery, one per GCP project. Each had its own JSON key file, its own set of IAM roles, and its own set of problems. When a local CLI tool needed to query production data, it used one key. When it needed to write to the raw data project, it used a different key. Our Slack bot had two keys mounted as secrets. The monitoring scripts had yet another. Every new tool or integration meant figuring out which key to use, and getting it wrong meant cryptic PERMISSION_DENIED errors that could take 20 minutes to debug.
Category:
Lessons Learned
DRY Revenue Logic: Extracting a Shared dbt Macro for Partner-Specific Calculations
by Marc
written by Marc
Revenue numbers that don’t add up are the kind of bug that erodes trust fast. Last month I tracked down a case where a quote adjustment of over two thousand euros was producing zero revenue in our reporting layer. The root cause? Duplicated business logic across two dbt models — logic that had drifted apart over time.
When Your Application Mutates created_at: Fixing Timestamps with CDC History
by Marc
written by Marc
The application was silently rewriting created_at after record creation, making child records appear older than their parents. Here is how I caught it and fixed it using raw CDC history in the dbt staging layer.
I spent two days hunting a revenue bug that produced numbers exactly 100 euros off on specific records. The SQL was correct. The data model was correct. The individual fields were correct. The bug was in how I was combining them — a subtle interaction between revision-aware bid fields and delta-based quote adjustment fields in a Unified Star Schema metrics bridge.
Our data team of four was drowning in ad-hoc requests. “What was the revenue last month?” “How many active customers do we have in Germany?” “What’s the conversion rate for Q4?” Each question meant someone opening a SQL editor, writing a query, formatting the results, and posting them in Slack. So I built an AI-powered data assistant: a Slack bot backed by Claude and BigQuery that lets anyone on the team ask business questions in natural language and get SQL-backed answers in seconds.
How two duplicate webhook events, arriving 2ms apart, broke 12 dbt tests in production — and the surprisingly simple fix that made the sync idempotent.
Older Posts