We had three Cloud Functions handling exchange rate data. One fetched rates from an external API and wrote them to Google Cloud Storage. A second read from GCS and loaded into BigQuery. A third was an abandoned earlier attempt that loaded directly from the API to BigQuery but had never been decommissioned. Three functions, three sets of logs, three potential failure points — for what is fundamentally one job: “get today’s exchange rates into BigQuery.”
Category:
Tools & Automation
We have an AI-powered Slack bot that answers business data questions. It uses a knowledge base — schema descriptions, business rules, SQL query patterns — to generate accurate queries. The problem: this knowledge base must stay in sync with two upstream repos. When a BI model changes, the bot needs to know about new dimensions and measures. When dbt models change, the bot needs an updated schema index. Keeping this in sync manually was a recipe for stale knowledge and wrong answers.
We lost ~25 webhook events in 7 minutes because our Cloud Run service was scaling to zero. Here is how I used Cloud Scheduler to toggle min-instances during business hours — and the workaround for Cloud Scheduler not supporting PATCH requests.
Our BI tool does not expose a usage API. So I built a headless Playwright scraper on Cloud Run that logs into the BI platform, scrapes dashboard usage metrics, loads them into BigQuery, and sends weekly Slack summaries. Here is how I handled the hardest part: session management.
I built a paper trading bot that runs on a Raspberry Pi 5, executes 8 strategies across stocks and crypto, and has been trading a simulated 10,000 euro portfolio since early 2026. This post covers the technical details: the strategy code, the backtester that validated the strategies against 195,000 candles, the yfinance pitfalls that cost me days of debugging, and the web dashboard that lets me monitor everything from my phone.
We had been paying roughly 400 euros a month for a managed ETL connector to sync our CRM data into BigQuery. Six tables, a few thousand records each, updated a handful of times per day. The connector worked, but it had problems: data landed in a US-region dataset (we needed EU for GDPR compliance), the CDC implementation had quirks that caused phantom duplicates, and every time we needed to debug something we were staring at a black box. So I replaced it with a custom Cloud Run service. The whole thing runs for under 5 euros a month.
Older Posts