Golden Dataset (AI Eval Set)

Concept

A golden dataset is a version-controlled set of question–SQL–answer triples with hand-verified correct results, used to score an AI analyst's accuracy on your own schema and definitions before and after any model, prompt or schema change.

Layman Explanation & Analogy

An exam paper for your AI with the answer key attached: 20–50 real business questions whose correct numbers a senior analyst has signed off. New model version? Sit the exam. Score up, ship it; score down, revert — no vibe-checking.

Worked real-world example

A 24-question suite scores 54% exact-match; 5 of 7 misses cluster on gross-vs-net trap columns. Three schema notes documenting net revenue fix 4 misses at once — the suite proved the team needed definitions, not a bigger model.

What people get wrong & common traps

Score the SQL as well as the number — a right number from wrong SQL is luck that fails next month. Re-verify answers quarterly (refunds and restatements rewrite history), retire questions every model aces, and promote every production wrong-answer into a new golden question.

Related terms