Seven Questions to Ask Before You Trust an AI's Answer About Your Data

An AI can write a SQL query that runs in milliseconds, produces zero error messages, returns a beautifully formatted number, and still be answering a completely different question than the one you asked.

This is the most dangerous failure mode in modern data analytics. When code fails with a red syntax error, you know it broke. But when an AI produces a semantically wrong query, it fails in complete silence — presenting an inaccurate number with 100% confidence.

Before you take a number generated by an AI assistant or text-to-SQL tool and paste it into a slide deck for your CEO or board, run through these seven sanity checks.

1. Does the SQL actually exist, or is the tool asking for blind trust?

The first filter is binary: did the AI show you the exact query it executed, or did it only give you a number?

A tool that shows only a summary number or a chart is asking for blind faith. If you cannot inspect the underlying SQL code, you cannot verify whether the calculation matches your company's definitions. A tool that displays the SQL behind every single number gives you, your analysts, and your engineers the ability to audit the logic in seconds.

2. Which table did it actually query? (The Schema Ambiguity Trap)

In real-world databases and CRMs, the word "revenue" or "orders" rarely lives in just one table. You might have:

• orders (gross order totals, including shipping and sales tax).
• order_line_items (individual product prices before discounts).
• stripe_transactions (net settled cash deposits after payment gateway fees).

-- ❌ The AI queries the raw orders table (Includes $15k in sales tax & shipping):
SELECT SUM(order_total) FROM orders WHERE status = 'paid';

-- ✅ The finance team expects Net Product Revenue:
SELECT SUM(subtotal_price - discount_amount) FROM order_line_items;

The Check: Look at the FROM clause immediately. Did the AI pick the table your team actually considers the source of truth for that metric?

3. What does the WHERE clause actually filter? (The SQL NULL Trap)

The WHERE clause is where "technically ran successfully" and "calculated the right number" most frequently diverge.

A classic example is handling cancelled orders. Many models write:

-- ⚠️ THE SQL NULL TRAP
SELECT SUM(order_total) 
FROM orders 
WHERE order_status != 'cancelled';

Why this breaks: In standard SQL three-valued logic, if an order is newly created and its order_status is NULL (unassigned), NULL != 'cancelled' evaluates to UNKNOWN, and that order is silently deleted from your report!

The Check: Verify that the WHERE clause:

1. Filters out refunds and cancellations explicitly.
2. Uses the correct date field (e.g. order_created_at vs shipped_at vs paid_at).
3. Handles NULL columns safely using COALESCE or IS NOT NULL.

4. What is the join key, and does it cause a fan-out duplication?

A join on a non-unique key is the #1 reason an AI-generated number looks mysteriously 2x to 5x higher than reality.

Layman Example: If you ask for "Revenue and ad spend by day," an AI might join your orders table directly to your ad_spend table on date:

-- ❌ DANGEROUS FAN-OUT JOIN:
-- If 5 different ad campaigns ran on Monday, EVERY order placed on Monday 
-- gets multiplied by 5 in the join!
SELECT 
  o.order_date, 
  SUM(o.order_total) AS multiplied_revenue
FROM orders o
JOIN campaign_spend s ON o.order_date = s.spend_date
GROUP BY o.order_date;

The Check: Did the AI aggregate each side of the join into a summary subquery or CTE before joining them on date?

5. Does the aggregation grain match what you asked?

"Revenue by campaign" and "Revenue by campaign by day, summed" can return different totals if a user's subscription touches multiple touchpoints over time.

The Check: Look at the GROUP BY clause. Are the columns in GROUP BY matching the exact entity you asked for (e.g. customer_id vs order_id vs session_id)?

6. Would a domain expert on your team recognize this business logic?

You do not need to be a software engineer to sanity-check an AI query. Simply read the English translation of what the query is doing:

"The AI took all rows from the raw signups table from last week, counted every row regardless of email verification, and divided by total site visits."

If you mention that description to your Head of Growth and they say "Wait, we only count verified business emails as signups," you just caught a critical discrepancy before it reached executive leadership.

7. How does it handle edge cases (Zeroes, Timezones, and Negatives)?

AI models tend to assume a clean, idealized world. Real database data is messy:

• Zero-Spend Days: Does the query use NULLIF(spend, 0) or SAFE_DIVIDE to prevent runtime crashes?
• Timezone Drift: If your ad platform is in Eastern Time and your database stores timestamps in UTC, did the query convert timezones before grouping by day?
• Refund Amounts: Are refunds stored as negative numbers in the orders table, or as positive numbers in a separate refunds table?

-- ✅ Clean edge-case handling:
SELECT 
  DATE(order_created_at, 'America/New_York') AS local_day,
  ROUND(SUM(net_revenue) / NULLIF(SUM(ad_spend), 0), 2) AS roas
FROM daily_metrics
GROUP BY 1;

The 60-Second Query Audit Flowchart

Why DataLens Always Ships the SQL

At DataLens, we believe that an analytics number without its accompanying SQL query is untrustworthy.

Every answer expands to show the exact query that produced it, and one click re-runs that query live to check the number hasn't drifted — so you get the speed of natural-language exploration with the rigor of verified analytics engineering, and never have to take a figure on faith.