Benchmarking an AI Analyst on Your Own 20 Golden Questions (fluency is not accuracy)
Chinmay Raibagkar · scores models on verified answers, not vibes📝 with a 20-question exam template, not a lecturePublished September 6, 2026
The 60-second version
- Fluency improves every release whether accuracy does or not — only a scored suite tells the difference.
- Twenty real questions with verified SQL and numbers beat a hundred synthetic ones; misses cluster in trap columns.
- Re-run on every change: new model, new prompt, new table — ship what moves the score, revert what does not.
"The AI feels smarter than last quarter." Does it actually answer better — on your schema, your definitions, your edge cases? Fluency improves with every model release whether accuracy does or not, and the only way to tell the difference is to stop vibe-checking outputs and start scoring them against questions whose answers you already know.
That scored set is a golden dataset: 20–50 canonical business questions with verified SQL and verified numbers. vibes are not a metric This post shows how to build one, what to score, and how to run it so model upgrades become measured decisions instead of mood readings.
An exam the AI cannot bluff
The Golden-Question Loop
Data JourneyCollect
Real questions your team asks, covering joins, filters, grains and known trap columns — each with a verified answer.
Score
Run each question through the tool, compare the returned number and SQL against verified truth. Fluency earns zero points.
Decide
New model, new prompt, new schema notes? Re-run the set. Ship what moves the score; revert what does not.
A golden dataset is a version-controlled set of question–SQL–answer triples that define correct behaviour for your data. It is the analytics equivalent of a software test suite: nobody would deploy code without tests, but teams deploy new models, prompts and schema changes against production questions daily with no suite at all.
Score the number, review the SQL. A correct number from wrong SQL is luck that will fail next month on different data; wrong SQL with a correct number scores zero. Both artefacts matter — the number for business trust, the SQL for durability.
Coverage over cleverness
Twenty well-chosen questions beat a hundred variations of "revenue last month". Cover the failure modes from the trust posts:
| Category | Example question | What it tests |
|---|---|---|
| Grain + dedup | "Orders and revenue by day last week" | Order-grain counting, no fan-out |
| Join path | "Spend vs delivered revenue by channel" | Spend→orders join at right grain |
| Business definition | "Active customers in August" | Resolves to the governed definition, not a guess |
| Filter correctness | "Prepaid-only ROAS, Maharashtra" | WHERE logic on real dimensions |
| Date math | "Trailing 7-day CAC vs prior 7 days" | Window arithmetic, timezone day cuts |
| Trap column | "Net revenue excluding tax and refunds" | Picks net over gross, excludes cancelled |
| Ambiguity | "How are we doing on Meta?" | Clarifies or states assumptions instead of guessing |
| Multi-source | "Blended MER with COD-adjusted revenue" | Cross-source reasoning, RTO awareness |
What a 24-question suite revealed
A rubric that survives gaming
Three Levels of Correct
Reporting HierarchyExact match
Number equals verified truth to the paisa. Required for definitional questions with deterministic answers.
Tolerance match
Within a stated band (±1–2%) for questions with legitimate variance: FX dates, window edges, rounding.
SQL-correct, number-off
Joins, filters and grain right; number differs for an explainable data reason (fresh partition, late refund).
Score zero for: correct-looking number from wrong SQL, unasked-for filters silently added, definitions invented rather than resolved, and confident answers to ambiguous questions with no stated assumption. That last one matters most — an AI that guesses instead of clarifying will eventually guess with your money.
Operating the suite
From Suite to Habit
Process FlowSeed with 20 questions from real history
Support tickets, Slack questions, board asks. Real beats synthetic — synthetic questions test imagination, real ones test the job.
Verify answers by hand, once
A senior analyst signs off each triple. The suite is only as golden as its verification; this is the expensive step and it happens once.
Run on every model, prompt or schema change
New model version, edited system prompt, new table: re-run before shipping. Regressions announce themselves as diffs.
Grow from failures, prune the trivial
Every production wrong-answer becomes a new golden question. Questions every model aces forever get retired.
Run the suite like this every model change →
- Seed 20 real questions — support tickets and board asks, each with a verified answer.
- Verify by hand, once — a senior analyst signs off every triple before it counts.
- Re-run on every change — new model, prompt, or table ships only if the score holds.
- Grow from failures — every production wrong answer becomes a new golden question.
Refresh answers, not just questions. Verified numbers decay as late-arriving data, refunds and restatements rewrite history. Re-verify the suite's answers quarterly — a suite asserting stale truth punishes correct models for being right.
Quick gut-check
One question on shipping models. score, do not vibe!
A new model feels smarter in the demo. Do you ship it?
Frequently Asked Questions
Isn't 20 questions too few to mean anything?
It is too few to rank models on a leaderboard and plenty to govern your deployment. The suite measures fitness for your schema and definitions, where failures cluster in a handful of known traps. Coverage of your traps matters; statistical power over trivia does not.
Who owns maintaining this?
The analyst who verifies the answers — typically whoever currently answers these questions for the business. Budget half a day to seed, an hour a quarter to refresh. The cost of not having it is every model change evaluated by anecdote.
Can DataLens-style tools with visible SQL use the suite directly?
Ideally — visible SQL makes scoring SQL-correctness fast (the 60-second audit becomes a 60-second grade). Black-box tools can only be scored on numbers, which hides the lucky-correct failure mode. Auditability is a prerequisite for meaningful evals, not a nice-to-have.
Summary & Next Steps
Fluency is not accuracy, and demos are not decisions. A golden dataset of verified question–SQL–answer triples converts "feels smarter" into a score, a diff, and a ship/no-ship call.
- Use a golden dataset to govern every model, prompt and schema change.
- Use schema mapping notes as the highest-ROI fix the suite will recommend.
- Use text-to-SQL audits to grade the SQL behind every scored number.
Takeaways for your next report
- Fluency is not accuracy — score the number, review the SQL.
- Twenty real questions beat a hundred synthetic ones.
- Re-run on every model, prompt, or schema change; ship the diff.
- Failures cluster in definitions — three schema notes beat a bigger model.
Chinmay Raibagkar
About author →Founder of DataLens AI. He helps non-technical teams read their ad and database numbers with confidence — which number to trust, what to do next, and what to ignore.