The Real Cost of an AI Analytics Question: Tokens, Query Bytes, and What Drives the Bill
The 60-second version
Two costs stack on every AI-generated answer, and the smaller one gets all the attention. A line-by-line breakdown of where the money actually goes.
- What happened, in one line
- What to do about it this week
- What you can safely ignore
"How much does an AI analytics question cost?"
Almost everyone answering that question reaches for token pricing, does some arithmetic, and lands on a fraction of a cent. That arithmetic is correct and it is usually the smaller half of the bill.
Two costs stack on every AI-generated answer: the model call and the query execution. The first is measured in tokens and is remarkably cheap. The second is measured in bytes scanned and can be a hundred times larger. Nearly all cost-control effort goes into the first one.
The two stacks
Where the Money Goes
Data JourneyModel call
Schema in, SQL out, plus a summary of the results. Priced per million tokens, with input far cheaper than output and caching cheaper still.
Query execution
The warehouse charges for data read. Independent of the model, the prompt, and how clever the question was.
The ratio
A well-filtered query keeps the two comparable. One unfiltered scan of a large table makes the model cost a rounding error.
Stack 1: The model call
A single question typically involves two or three model calls:
| Call | Input | Output | Notes |
|---|---|---|---|
| Table selection (if used) | Table names + one-line descriptions | A shortlist | Small, cheap, often skipped on small schemas |
| SQL generation | Selected schema + question + system prompt | A SQL query | The main input cost — schema dominates |
| Result summarisation | Result rows + question | Prose answer | Output-heavy; result size controls it |
Input tokens dominate, and within input, schema dominates. A generous schema block for a dozen tables with descriptions runs to a few thousand tokens; the question itself is perhaps thirty. Everything else is noise.
Which gives the three levers, in order of effect:
Constrain schema, don't just cache it
Sending 200 tables when 8 would do multiplies input cost by roughly the same factor — and, as covered in preventing hallucinated columns, makes accuracy worse by offering deprecated and near-duplicate tables alongside the current ones. This is the rare optimisation that improves cost and quality together.
Cache the stable prefix
Every major provider offers prompt caching: the unchanging prefix — system prompt plus schema — is charged at a large discount on subsequent calls within the cache window. Since the schema is the bulk of the input and it changes rarely, this is close to free money for any conversation with more than one turn.
The structural requirement is that the cacheable content comes first and is byte-identical. Putting the user's question before the schema defeats caching entirely, because the prefix differs on every call.
Cap the result rows sent back for summarisation
A query returning 50,000 rows does not need 50,000 rows in the summarisation call. Send the top N plus aggregate statistics. This is the one place output-heavy costs creep in unnoticed, and truncation costs nothing in answer quality — nobody was going to read row 4,000.
Order of magnitude, not a quote: at current frontier-model pricing, a question with a few thousand tokens of schema in and a few hundred out lands well under a cent. Cached, it is a fraction of that. This is why token cost is rarely the thing that matters — and why the effort spent optimising it is usually misallocated.
Stack 2: The query
This is where the real variance lives, and where a generated query is riskier than a human-written one — because the thing that wrote it has no memory of last month's bill.
The same question, four ways
The three controls that matter
Dry-run every query before executing it. A dry run returns totalBytesProcessed for free in under a second. Any tool generating SQL can show you what a query will cost before running it — and one that does not is choosing not to do the cheapest safety step available.
Set maximumBytesBilled on every execution. A hard ceiling means a runaway query fails instead of billing. DataLens passes this on generated queries as a second guard behind the dry run.
Point the tool at aggregates, not raw tables. The single largest structural lever. If the AI can only reach a semantic schema of pre-aggregated, partitioned tables, the 2.4 TB scan is not reachable — not because the model behaved, but because the data is not there to scan.
Show query
What actually drives your monthly bill
Neither stack, per question, is the interesting number. Volume is.
The Four Multipliers, In Order of Effect
Process FlowScheduled reports
A daily report is 30 questions a month whether or not anyone reads it. A report per campaign per day, across 40 campaigns, is 1,200 — and this is the line that grows without anyone deciding to grow it.
Retries and repair loops
Every failed generation costs a full model call. An uncapped repair loop can triple the model cost of a single question. Cap it at two.
Conversation depth
Turn five carries turns one through four in context. Without prompt caching, cost per turn grows with conversation length; with it, it barely moves.
Exploratory query cost
A user refining a question runs five near-identical queries. If each scans 18 GB, that is $0.55 of exploration on one question — which is fine, and worth knowing.
Scheduled reports are the line that surprises people. An ad-hoc question happens when someone asks it. A scheduled report happens 30 or 730 times a month regardless. Both cost stacks multiply, and unlike a dashboard nobody opens, nothing about a scheduled report gets cheaper when it is ignored. Audit the schedule quarterly and delete what nobody reads.
Tracking it
If you are running this at any scale, log both stacks per question:
Show query
Read query_to_model_ratio. Below about 5, your queries are well-controlled and further optimisation should target the model stack. Above 20, stop tuning prompts — the money is in the warehouse, and one materialised rollup will save more than every prompt change you could make.
The comparison people actually want
Set against the alternative, both numbers are small.
Cost per answered question
Frequently asked questions
Which is bigger, tokens or bytes?
Almost always bytes, and usually by an order of magnitude or more — unless your warehouse is already well-organised with pre-aggregated tables, in which case they become comparable. Measure the ratio rather than assuming; it tells you which lever to pull.
Does prompt caching really help?
Substantially, for multi-turn conversations, because the schema block is the bulk of the input and it is identical every turn. The requirement is structural: the cacheable prefix must come first and be byte-identical. Any per-request variable placed before the schema silently defeats it.
Should I use a cheaper model for SQL generation?
Worth testing, and the trade-off is not the obvious one. A cheaper model that generates a wrong query costs you a repair loop — two more model calls — and possibly a wasted query execution. Where the query stack dominates, the cost difference between models is often smaller than the cost of one extra failed attempt.
How do I stop a user from running an expensive query?
Three layers, all cheap: maximumBytesBilled as a hard ceiling, a dry-run estimate shown before execution, and constraining the AI to aggregate tables so the expensive scan is not reachable. The third is the one that works without anyone having to make a good decision.
Do scheduled AI reports cost more than dashboards?
Per refresh, a scheduled AI report costs a model call plus a query, while a dashboard tile costs a query. So slightly more, and the difference is usually swamped by how many tiles a dashboard has and how often it auto-refreshes. A single daily narrative is frequently cheaper than a dashboard with twenty tiles refreshing hourly.
The summary
- Two stacks: model call (tokens) and query execution (bytes). The second is usually the larger, often by a lot.
- Model cost is dominated by schema in the input. Constrain the schema — it cuts cost and improves accuracy at the same time — and cache the stable prefix.
- Query cost varies by thousands of times for the same question, depending entirely on what it scans. Dry-run before executing, cap with
maximumBytesBilled, and point the tool at aggregates so the expensive scan is unreachable. - Volume drivers, in order: scheduled reports, uncapped repair loops, conversation depth, exploratory iteration.
- Log both stacks and watch the ratio. Above 20:1, stop tuning prompts — the money is in the warehouse.
BigQuery Cost Estimator
Estimate what a query will cost before you run it — from validator bytes, with a table-scan simulator and a paste-your-query cost audit. On-demand BigQuery pricing, made concrete.
Chinmay Raibagkar
About author →Founder of DataLens AI. He helps non-technical teams read their ad and database numbers with confidence — which number to trust, what to do next, and what to ignore.