Why Fine-Tuning LLMs for Internal SQL Is Almost Always a Mistake (retrieve schema, don’t bake it)
Published September 17, 2026
The 60-second version
- Fine-tuned models bake schema into frozen weights. The moment an engineer renames a column or adds a status code, the model hallucinates stale syntax.
- Dynamic context injection (RAG) is instant and testable. Feeding current table DDLs and definitions in the prompt outperforms fine-tuning at zero retraining cost.
- The fix: use frontier general models, keep schema definitions in a version-controlled catalog, and test with golden evaluation questions.
An engineering team collects 5,000 internal SQL queries, cleans them into pairs of English questions and database code, and spends ₹8,00,000 in GPU compute fine-tuning an open-source model. On day one, the demo is dazzling.
On day forty-five, an engineer renames orders.gross_amount to orders.gross_total and deprecates the legacy_status column.
The custom model immediately starts producing queries that fail execution, or worse, queries that run silently using deleted columns. The team realizes they now own a permanent retraining pipeline.
The mistake was not the model choice. It is that fine-tuning bakes living schema definitions into frozen matrix weights. A company database is a living organism — columns change, business definitions evolve, and table relations shift weekly. Attempting to solve a data retrieval problem by retraining model parameters is the most expensive way to build technical debt.
The seductive promise of the custom model
🤖 'Our own private model'
Train once, know everything. The intuition sounds logical: feed the model thousands of historical queries so it learns your internal shorthand, tables, and company dialect.
- Feels proprietary and defensible
- Zero prompt latency overhead
- Sounds great in quarterly engineering reviews
⚙️ Static weights, dynamic schema
Weights are frozen in time; databases change daily. The model cannot distinguish between a query written three years ago with outdated logic and current best practice.
- Breaks at every schema migration
- Retraining costs thousands of dollars
- Hallucinates deprecated business logic
Fine-tuning teaches style and formatting, not live business context. A database requires fresh facts, not memorized habits.
When an LLM writes bad SQL, it is rarely because it does not know the syntax of a LEFT JOIN or a GROUP BY. Modern general-purpose frontier models already know SQL syntax exceptionally well.
The failure mode is almost always semantic ignorance: not knowing which of four tables named customers holds the verified active list, or whether revenue includes or excludes shipping tax. Fine-tuning attempts to memorize that knowledge into model weights where it can neither be audited nor easily edited.
Four structural traps of fine-tuning for SQL
If you choose the fine-tuning route for internal analytics, you will inevitably run into four walls.
The Historical Poisoning Trap
Historical company queries are full of bad SQL, hotfixes, and deprecated joins. Training on history teaches the model bad habits that took your data team two years to unlearn.
The Migration Amnesia
When a table schema changes, prompt-based systems update in 5 seconds via a YAML catalog. A fine-tuned model requires gathering new data and kicking off a GPU cluster run.
The Model Upgrade Treadmill
Every 4 months, frontier general models leap in capability. If your architecture relies on fine-tuning an older base model, you are perpetually stuck behind the state of the art.
Weights cannot be inspected. When a fine-tuned model writes WHERE status = 'PAID' instead of WHERE status = 'COMPLETED', you cannot open a config file and fix it. You have to re-weight your training dataset, train another checkpoint, and pray the update doesn't degrade performance on 20 other queries.
The dynamic context alternative
The architectural pattern that consistently wins in production is Dynamic In-Context Schema Retrieval (Schema RAG).
Instead of trying to teach the model your database permanently, you keep the model completely vanilla and feed it the exact context it needs at the moment the question is asked. retrieve schema, don't bake it
The Production Text-to-SQL Stack
Reporting Hierarchy1. Semantic Catalog (Human Truth)
A version-controlled YAML or Markdown directory defining table descriptions, verified join keys, and business metrics. Changes here take effect immediately.
2. Context-Aware Retrieval (Narrowing the World)
When a user asks about 'churned subscribers', search the catalog and inject only the 3 relevant table definitions into the prompt. Never pass a 100-table database dump.
3. Frontier General LLM + Dry-Run Guardrail
A standard frontier model generates SQL using the injected DDL. BigQuery dry-runs the query to validate syntax and estimate bytes before returning results.
If someone renames a column on Tuesday morning, an analyst updates one line in the table catalog. By Tuesday 9:01 AM, every generated query is using the new column. Total cost: ₹0. Training time: 0 seconds.
A side-by-side scorecard: fine-tuning vs retrieval
Consider what happens to accuracy over a 60-day period as your engineering team ships new features and alters database schemas.
| Capability | Fine-Tuned Custom Model | Dynamic Schema RAG |
|---|---|---|
| Adapting to schema changes | Weeks (retrain & re-evaluate) | Seconds (edit catalog file) |
| Setup & maintenance cost | High (GPU compute + data cleaning) | Low (plain documentation) |
| Auditability | Black box (cannot see why weights chose a column) | Transparent (prompt shows exact DDL used) |
| Model portability | Locked to base model architecture | Instant swap to any frontier LLM |
| Risk of silent failure | High (outputs outdated columns confidently) | Low (compiler catches syntax mismatches) |
How to build your text-to-SQL architecture do these four steps instead →
- Extract minimal DDLs — document table names, column types, and foreign key relationships for your top 10 reporting tables.
- Annotate trap columns — explicitly write down definitions like: 'use `created_at` in IST, never `system_ingested_at`'.
- Dry-run before execution — compile SQL against BigQuery's API without running it to catch missing columns instantly.
- Maintain 20 golden test questions — run your benchmark suite weekly to ensure accuracy never regresses.
Quick gut-check
One question. Get it and the whole post clicks. 30 seconds, no maths!
Why does dynamic schema injection (passing DDL in prompts) beat fine-tuning for enterprise SQL generation?
Frequently asked questions
Is there ever a scenario where fine-tuning makes sense?
Yes, but for dialect translation, not schema memory. If you built a proprietary, custom query language or heavily restricted DSL (Domain-Specific Language) that general LLMs have never seen in their public training corpus, fine-tuning teaches the grammar. But for BigQuery, Snowflake, Postgres, or MySQL, general LLMs already speak the language fluently.
What if my database has 400 tables and won't fit in the prompt?
You should never pass 400 tables to an LLM. No human analyst queries 400 tables at once. Use a lightweight semantic routing layer: match the user's question to the 3–5 relevant tables (e.g. orders, customers, campaign_spend), and pass only those 5 table definitions into the context window.
Doesn't fine-tuning protect private customer data better?
No. In fact, fine-tuning often creates security vulnerabilities because customer PII can accidentally get baked into model weights, where it is vulnerable to prompt extraction attacks. Keeping models stateless and querying views with redacted PII is vastly safer.
How does DataLens approach this?
DataLens connects directly to your BigQuery schemas and ad APIs dynamically. It reads current table structures, applies dry-run validations, and inspects errors in real-time. If you add a column to your table today, DataLens can query it immediately without a retraining step.
The summary
- Fine-tuning for text-to-SQL confuses style with facts. LLMs already know SQL syntax; what they lack is your company's current schema.
- Baking database schemas into model weights guarantees obsolescence the moment an engineer alters a table.
- Dynamic schema retrieval (injecting current table definitions into the prompt) is transparent, instant to update, and costs zero training compute.
- Always validate generated SQL with database dry-runs before execution to catch typos and syntax mismatches automatically.
- Maintain a version-controlled dictionary of business definitions; let the model generate the code, but let your team govern the meaning.
Takeaways for your next report
- Fine-tuning bakes living database schemas into frozen model weights that rot over time.
- General-purpose frontier models already know SQL syntax; they only need your current schema context.
- Dynamic context injection (RAG) updates in seconds via plain documentation with zero retraining.
- Dry-run compilation and golden question suites are the true guardrails for production AI analytics.
Chinmay Raibagkar
About author →Founder of DataLens AI. He helps non-technical teams read their ad and database numbers with confidence — which number to trust, what to do next, and what to ignore.