The Real Cost of an AI Analytics Question (tokens are the smaller half)
Chinmay Raibagkar · has seen a missing WHERE clause cost $15💸 with a cost-of-a-question sandbox, not a lecturePublished August 28, 2026
The 60-second version
- Two bills stack on every answer: the model call in tokens and the query in bytes scanned.
- The query usually dominates — the same question costs 6,000x more against raw tables than against a rollup.
- Control it structurally: dry-run everything, cap bytes billed, and point the tool at aggregates.
"How much does an AI analytics question cost?"
Almost everyone answering that question reaches for token pricing, does some arithmetic, and lands on a fraction of a cent. That arithmetic is correct and it is usually the smaller half of the bill. spoiler: it is the bytes
Two costs stack on every AI-generated answer: the model call and the query execution. The first is measured in tokens and is remarkably cheap. The second is measured in bytes scanned and can be a hundred times larger. Nearly all cost-control effort goes into the first one.
The two stacks
Where the Money Goes
Data JourneyModel call
Schema in, SQL out, plus a summary of the results. Priced per million tokens, with input far cheaper than output and caching cheaper still.
Query execution
The warehouse charges for data read. Independent of the model, the prompt, and how clever the question was.
The ratio
A well-filtered query keeps the two comparable. One unfiltered scan of a large table makes the model cost a rounding error.
Stack 1: The model call
A single question typically involves two or three model calls:
| Call | Input | Output | Notes |
|---|---|---|---|
| Table selection (if used) | Table names + one-line descriptions | A shortlist | Small, cheap, often skipped on small schemas |
| SQL generation | Selected schema + question + system prompt | A SQL query | The main input cost — schema dominates |
| Result summarisation | Result rows + question | Prose answer | Output-heavy; result size controls it |
Input tokens dominate, and within input, schema dominates. A generous schema block for a dozen tables with descriptions runs to a few thousand tokens; the question itself is perhaps thirty. Everything else is noise.
Which gives the three levers, in order of effect:
Constrain schema, don't just cache it
Sending 200 tables when 8 would do multiplies input cost by roughly the same factor — and, as covered in preventing hallucinated columns, makes accuracy worse by offering deprecated and near-duplicate tables alongside the current ones. This is the rare optimisation that improves cost and quality together.
Cache the stable prefix
Every major provider offers prompt caching: the unchanging prefix — system prompt plus schema — is charged at a large discount on subsequent calls within the cache window. Since the schema is the bulk of the input and it changes rarely, this is close to free money for any conversation with more than one turn.
The structural requirement is that the cacheable content comes first and is byte-identical. Putting the user's question before the schema defeats caching entirely, because the prefix differs on every call.
Cap the result rows sent back for summarisation
A query returning 50,000 rows does not need 50,000 rows in the summarisation call. Send the top N plus aggregate statistics. This is the one place output-heavy costs creep in unnoticed, and truncation costs nothing in answer quality — nobody was going to read row 4,000.
Order of magnitude, not a quote: at current frontier-model pricing, a question with a few thousand tokens of schema in and a few hundred out lands well under a cent. Cached, it is a fraction of that. This is why token cost is rarely the thing that matters — and why the effort spent optimising it is usually misallocated.
Stack 2: The query
This is where the real variance lives, and where a generated query is riskier than a human-written one — because the thing that wrote it has no memory of last month's bill.
The same question, four ways
The three controls that matter
Dry-run everything
totalBytesProcessed comes back free in under a second. Show the cost before running, not on the invoice.
Cap bytes billed
maximumBytesBilled on every execution turns a runaway query into a failure instead of a bill.
Point at aggregates
If the AI can only reach pre-aggregated, partitioned tables, the 2.4 TB scan is simply unreachable.
What drives your monthly bill
Neither stack, per question, is the interesting number. Volume is.
The Four Multipliers, In Order of Effect
Process FlowScheduled reports
A daily report is 30 questions a month whether or not anyone reads it. A report per campaign per day, across 40 campaigns, is 1,200 — and this is the line that grows without anyone deciding to grow it.
Retries and repair loops
Every failed generation costs a full model call. An uncapped repair loop can triple the model cost of a single question. Cap it at two.
Conversation depth
Turn five carries turns one through four in context. Without prompt caching, cost per turn grows with conversation length; with it, it barely moves.
Exploratory query cost
A user refining a question runs five near-identical queries. If each scans 18 GB, that is $0.55 of exploration on one question — which is fine, and worth knowing.
Scheduled reports are the line that surprises people. An ad-hoc question happens when someone asks it. A scheduled report happens 30 or 730 times a month regardless. Both cost stacks multiply, and unlike a dashboard nobody opens, nothing about a scheduled report gets cheaper when it is ignored. Audit the schedule quarterly and delete what nobody reads.
Ad-hoc plus scheduled, together
Your dry-run average — raw tables scan far more
Queries are well-controlled. The interesting comparison remains the question you would otherwise never have asked.
Tame the four multipliers biggest first →
- Audit the schedule quarterly — every unread daily report is 30 paid questions a month.
- Cap repair loops at two — every failed generation costs a full model call.
- Cache the stable prefix — schema first, byte-identical, or multi-turn costs balloon.
- Expect exploratory iteration — five near-identical queries per question is normal; make each cheap.
Tracking it
If you are running this at any scale, log both stacks per question:
Read query_to_model_ratio. Below about 5, your queries are well-controlled and further optimisation should target the model stack. Above 20, stop tuning prompts — the money is in the warehouse, and one materialised rollup will save more than every prompt change you could make.
The comparison you actually want
Set against the alternative, both numbers are small.
Cost per answered question
Quick gut-check
One question on where the money actually goes. follow the bytes!
Same question, same tokens — $0.0025 one way, $15 the other. What differed?
Frequently asked questions
Which is bigger, tokens or bytes?
Almost always bytes, and usually by an order of magnitude or more — unless your warehouse is already well-organised with pre-aggregated tables, in which case they become comparable. Measure the ratio rather than assuming; it tells you which lever to pull.
Does prompt caching really help?
Substantially, for multi-turn conversations, because the schema block is the bulk of the input and it is identical every turn. The requirement is structural: the cacheable prefix must come first and be byte-identical. Any per-request variable placed before the schema silently defeats it.
Should I use a cheaper model for SQL generation?
Worth testing, and the trade-off is not the obvious one. A cheaper model that generates a wrong query costs you a repair loop — two more model calls — and possibly a wasted query execution. Where the query stack dominates, the cost difference between models is often smaller than the cost of one extra failed attempt.
How do I stop a user from running an expensive query?
Three layers, all cheap: maximumBytesBilled as a hard ceiling, a dry-run estimate shown before execution, and constraining the AI to aggregate tables so the expensive scan is not reachable. The third is the one that works without anyone having to make a good decision.
Do scheduled AI reports cost more than dashboards?
Per refresh, a scheduled AI report costs a model call plus a query, while a dashboard tile costs a query. So slightly more, and the difference is usually swamped by how many tiles a dashboard has and how often it auto-refreshes. A single daily narrative is frequently cheaper than a dashboard with twenty tiles refreshing hourly.
The summary
- Two stacks: model call (tokens) and query execution (bytes). The second is usually the larger, often by a lot.
- Model cost is dominated by schema in the input. Constrain the schema — it cuts cost and improves accuracy at the same time — and cache the stable prefix.
- Query cost varies by thousands of times for the same question, depending entirely on what it scans. Dry-run before executing, cap with
maximumBytesBilled, and point the tool at aggregates so the expensive scan is unreachable. - Volume drivers, in order: scheduled reports, uncapped repair loops, conversation depth, exploratory iteration.
- Log both stacks and watch the ratio. Above 20:1, stop tuning prompts — the money is in the warehouse.
Takeaways for your next report
- Two stacks: tokens (fractions of a cent) and bytes scanned (dollars) — the second usually dominates.
- Same question, 6,000x cost spread — cost control is warehouse engineering, not prompt engineering.
- Dry-run everything, cap bytes billed, point the tool at aggregates.
- Watch volume: scheduled reports multiply everything, read or not.
BigQuery Cost Estimator
Estimate what a BigQuery query will cost before you run it, from validator bytes, with a table-scan simulator and a paste-your-query cost audit.
Chinmay Raibagkar
About author →Founder of DataLens AI. He helps non-technical teams read their ad and database numbers with confidence — which number to trust, what to do next, and what to ignore.