What an AI Analyst Can't Do Yet — an Honest List

I build one of these. This is the list I would want if I were evaluating one.

There is a genre of AI analytics marketing that implies the analyst role is solved — ask a question, get an answer, disband the data team. It is not solved, and the gap between what these tools do well and what they are sold as doing is where trust gets destroyed. A tool that is honest about its ceiling is more useful than one that is confidently wrong at the edge of it.

Eight things an AI analyst genuinely cannot do yet. For each: why the limit exists, and whether it is shrinking.

---

1. It cannot know what your columns mean

Schema retrieval gives a model table names, column names, types, and — if someone bothered to write them — descriptions. It does not give it the institutional knowledge that determines whether an answer is right.

Your orders table has total, subtotal, net_total and amount_paid. Which is revenue? The answer depends on whether tax is included, whether total was the pre-2024 field that stopped being maintained, and whether amount_paid is null for COD orders. No amount of reading the schema reveals any of that.

2. It cannot tell you the data is wrong

An AI analyst queries what is there. If your ad spend pipeline silently stopped ingesting on the 14th, it will confidently report that spend fell 60% — because in the data, it did.

Humans catch this through a mechanism that is hard to replicate: a felt sense that a number is implausible for this business, formed from months of watching it. A model has no prior over your business. It has a prior over businesses, which is far too weak to catch a 60% drop that is technically possible.

Anomaly detection helps at the margin, and it produces its own false positives — a genuine Diwali spike looks exactly like a data error to a statistical test.

3. It cannot design an experiment or establish causation

Ask "did the campaign work?" and you will get a correlation dressed in careful language. Causation requires a counterfactual: what would have happened without the campaign. That does not exist in your data because it never happened.

4. It cannot make a judgement call about what matters

"Should we cut this campaign?" depends on your cash position, your growth targets, whether the CEO promised the board a number, whether this channel is strategically important beyond its ROAS, and whether the team has bandwidth for the alternative. None of that is in the warehouse.

An AI analyst can tell you the campaign is at 0.8x contribution ROAS. Whether that means "cut it" or "it is a deliberate investment we agreed to fund for two quarters" is a business decision with inputs the tool cannot see.

5. It cannot maintain state across your organisation

Ask the same question in two conversations and you may get two SQL queries with slightly different filters. Ask "revenue" on Monday and it might include tax; ask on Thursday after a differently-phrased question and it might not.

This is the single largest practical objection to putting an AI analyst in front of an executive. A dashboard is at least consistently wrong. An AI analyst can be inconsistently right, which is worse — two people quote different numbers for the same metric in the same meeting, and trust in the whole system evaporates.

6. It cannot reliably do multi-step analysis with judgement at each step

"Find out why CAC went up last month" is not one query. It is: check whether spend rose or customers fell, then split by channel, then check whether the worsening channel had a CPM increase or a conversion-rate decrease, then check whether that coincided with a creative change or an audience change, then decide which thread to pull.

Each step requires deciding what to look at next based on what the previous step returned. Current tools do some of this and lose the thread on longer chains — an early wrong turn compounds, because nothing in the loop steps back and asks whether the whole line of inquiry is still sensible.

Is it shrinking? Yes, noticeably. This is the area where capability has moved most in the last two years. It is not solved, and long chains still need a human checking the branch points.

7. It cannot access what is not connected

Obvious, and constantly forgotten in practice. If your offline sales are in a spreadsheet, your influencer payouts are in someone's inbox, and your RTO data is in a courier portal nobody exported, the AI analyst's answer about total marketing efficiency is wrong — and it will not say so, because it has no way to know those things exist.

This is the most common source of confidently wrong answers in practice, and it has nothing to do with the model. A human analyst asks "is that everything?" A tool assumes what it can see is what there is.

8. It cannot take responsibility

When a number in a board deck is wrong, someone has to own it. That accountability is not a technicality — it changes behaviour. An analyst who will be asked about a number checks it differently from one who will not.

An AI analyst can show its SQL, its assumptions, and its row counts, so a human can take responsibility. But the checking still has to happen, and the tool's job is to make it cheap rather than to remove it.

---

What this means for evaluating one

---

The honest positive case

Having listed the ceiling, the floor is worth stating too, because it is higher than the sceptical version suggests.

An AI analyst removes the queue. The value is not that it replaces an analyst; it is that a question worth ₹200 of someone's attention gets asked at all, instead of being dropped because it was not worth interrupting a person for. Most organisations have far more questions than analyst-hours, and the questions that get dropped are not randomly selected — they are the small, exploratory, might-be-nothing ones, which is where a disproportionate share of useful findings live.

It is also genuinely good at the mechanical parts: recalling schema across dozens of tables, writing correct SQL syntax, applying a stated definition consistently within one conversation, and doing it in seconds. Those are real, and they are most of the volume of analytics work even if they are not most of its value.

---

Frequently asked questions

Will better models fix these?

Some of them. Multi-step reasoning (6) is improving quickly. Consistency (5) is being solved by architecture rather than model capability. Causation (3) and responsibility (8) are not model problems at all and will not be fixed by a larger one. Data quality (2) and unconnected data (7) are your problems, not the model's.

Should I not use one, then?

Use one, with the ceiling in view. The failure mode is not using the tool — it is treating its output as an answer rather than as a draft with its working shown. Every limit above is manageable if you know it is there.

What is the single highest-value thing I can do to make one work better?

Write column descriptions and define your core metrics somewhere deterministic. It is unglamorous and it addresses limits 1 and 5 directly — the two that cause the most wrong answers in practice.

How do I stop it inventing column names?

That one is a solved engineering problem rather than a capability limit — retrieval, constrained context and dry-run validation before execution. It gets its own post.

Does showing the SQL actually help non-technical users?

More than expected. Most people cannot write SQL and can read enough of it to spot "this only covers August" or "this is missing the refund filter" — which are the errors that matter. The value is not in reading the query; it is that the assumptions become visible at all.

---

The summary

Eight limits, and which are moving:

1. Does not know what your columns mean — shrinking, through documentation, not models.
2. Cannot tell you the data is wrong — partially addressable with monitoring.
3. Cannot establish causation — not shrinking; needs experimental design.
4. Cannot make business judgement calls — not shrinking; the inputs are not in the warehouse.
5. Cannot maintain consistent definitions alone — shrinking, via semantic layers.
6. Loses the thread on long multi-step analysis — shrinking noticeably.
7. Cannot see unconnected data — your problem, and the most common cause of confidently wrong answers.
8. Cannot take responsibility — structural.

The tool's job is to make checking cheap, not to remove the need for it. Evaluate one on whether it shows its SQL, admits ignorance, stays consistent, and tells you the cost before it runs.