Stale dashboards lie confidently (freshness checks that catch it first)
Chinmay Raibagkar · has debated a demand collapse that was a credential expiry🚨 with a three-layer validation suite, not a lecturePublished September 6, 2026
The 60-second version
- A broken pipeline looks exactly like a demand collapse. Nothing distinguishes “no new rows” from “no new orders” without checks.
- Three layers: freshness (did it arrive), shape (did it arrive whole), reconciliation (is it true).
- Separate lateness from wrongness in every alert, and route both somewhere a human actually reads.
Nobody noticed the pipeline had been broken for six days. The dashboard kept rendering — cheerfully, confidently — with last Tuesday's data relabelled as this Tuesday's, because nothing in the system distinguished "no new rows" from "no new orders." The Monday business review debated a demand collapse that was, in fact, a credential expiry.
Stale dashboards are worse than no dashboards: they convert a pipeline bug into a business decision. The fix is a small suite of freshness and validation queries — row-count deltas, null-rate guards, spend-vs-API reconciliation — run on a schedule, read by a human. This post builds that suite. confident and wrong is the worst kind of dashboard
Trust, but verify on a schedule
The Validation Suite
Data JourneyFreshness
When did each table last grow? A table whose max timestamp stops advancing is either broken or finished — and pipelines are never finished.
Shape
Row counts, null rates and distinct-key counts per day. A 90% volume drop or a suddenly-null column announces itself here.
Reconciliation
Warehouse spend vs platform API, order counts vs store admin. Two independent systems agreeing is the only real proof.
A freshness check answers "how old is the newest data I can query?" Data freshness is the underlying property; the check is the query that measures it. Everything below is designed to run as scheduled SQL with results routed somewhere a human actually reads — an email, a Slack channel, a morning report. A check nobody reads is documentation, not monitoring.
⏳ Late
Wait or fix the sync. Meta spend for yesterday is missing — a token expired, a transfer slipped. Response: re-auth, backfill, move on.
- Newest timestamp stopped advancing
- Known SLA per source
- Backfill recovers it fully
🔍 Wrong
Investigate now. Meta spend for yesterday is half the API — a timezone changed, an account dropped. Response: find the defect today.
- Data arrived but disagrees
- No backfill fixes a wrong join
- Needs a human before standup
One alert text for both trains people to ignore both. lateness waits, wrongness pages
Layer 1: did the data arrive?
The cheapest possible check, and the one that would have caught the six-day outage in the intro:
Set per-table thresholds from the source's nature, not from optimism: an intraday export lagging 6 hours is late; a daily-transfer table lagging 6 hours is normal. Document the SLA next to the threshold so the on-call human knows which alerts can wait for the scheduled sync.
Layer 2: did it arrive whole?
Fresh data can still be broken data: a schema change that nulls a column, a filter that drops 90% of rows, a backfill that doubles them. Shape checks compare today against recent history:
Three alerts, three different mornings
Layer 3: is it true?
Two systems that agree were probably both right; one system alone is a claim. The highest-value reconciliations in marketing data:
Spend vs platform API
Within 2–3% is healthy timing edges and timezone cuts. Beyond 5% is a real defect — a missed account, a changed timezone, a dropped date range.
Orders vs store admin
Exact-match territory. Any gap between warehouse orders and store admin is a pipeline bug, full stop — no tolerance band.
Attributed share vs history
The fraction of orders carrying a click ID should move slowly. A cliff means capture plumbing broke, not that customers changed.
Reconcile at the grain the business decides on. Daily reconciliation in reporting currency and reporting timezone — the normalisation layer from the companion post. Reconciling raw UTC-day API numbers against IST-day warehouse numbers manufactures false alarms that teach everyone to mute the channel.
Operate without alert fatigue
From queries to a habit one layer per week →
- Start with freshness on money tables — spend and orders, daily. Two queries catch the catastrophic class of outage; everything else is refinement.
- Add shape guards with median bands — fixed thresholds rot as the business grows, trailing-median bands grow with it. Review band widths quarterly.
- Add one reconciliation per source pair — spend vs API, orders vs admin. The checks that catch silent wrongness, one query each.
- Route to a read channel with an owner — a Slack channel with a named owner beats email. Unowned alerts become wallpaper within a month.
Frequently Asked Questions
Won't these checks cost a fortune to run on big tables?
No — they are aggregation queries over partitioned recent data: a day or two of partitions, a few columns, typically megabytes scanned. The suite's daily cost is ordinarily cents. If a check scans the full table history, restrict it to the trailing window — history does not change shape overnight.
How is this different from dbt tests or vendor observability?
It is the same instinct in scheduled-SQL form: for teams without a transformation framework or a data-observability vendor, these queries are the test suite. If you later adopt dbt, each check maps directly onto a test or snapshot — nothing is wasted.
What should the alert say to be actionable at 9 AM?
Table, layer (freshness/shape/recon), observed vs expected, and the documented runbook link. "raw_ad_spend freshness: newest event 2 days ago (SLA: daily) — runbook: check token, then backfill" resolves before standup. "Data issue detected" resolves never.
One check that the suite landed. answer before Monday, not during it
Monday's dashboard shows demand collapsed over the weekend. raw_ad_spend last advanced six days ago. What is your first conclusion?
Summary & Next Steps
Dashboards do not go stale loudly; they go stale confidently. A three-layer suite — freshness (did it arrive), shape (did it arrive whole), reconciliation (is it true) — converts silent pipeline rot into a morning message with an owner.
- Use freshness checks on every money table, thresholded by source SLA.
- Use deduped views so backfill recoveries do not create double-count incidents on the way past.
- Use freshness literacy in reporting so provisional windows are labelled, not debated.
Takeaways for your next report
- Dashboards go stale confidently, not loudly — a frozen pipe looks exactly like a demand collapse without checks.
- Run three layers: freshness (did it arrive), shape (did it arrive whole), reconciliation (is it true).
- Separate lateness from wrongness in every alert, threshold shape on trailing medians, and route everything to an owned channel.
- Start with freshness on money tables; each layer is one scheduled query a human actually reads.
BigQuery Cost Estimator
Estimate what a BigQuery query will cost before you run it, from validator bytes, with a table-scan simulator and a paste-your-query cost audit.
Chinmay Raibagkar
About author →Founder of DataLens AI. He helps non-technical teams read their ad and database numbers with confidence — which number to trust, what to do next, and what to ignore.