Agentic Analytics: Teaching an AI Analyst to Say I Don't Know

The dangerous thing an AI analyst does is not getting an answer wrong. It is getting one wrong in a way that looks completely right.

Not a hallucination, nothing so obvious. A real number, correctly computed, with a clean audit trail behind it. It just answers a slightly different question from the one you asked. A different time window. A different population. A metric that means almost the thing you meant.

Nothing on the screen marks it. No error, no warning, nothing missing from the trace. It looks exactly like the answer you wanted, because in every respect except the one that matters, it is one.

So it goes in the deck. Someone decides something with it. Nobody ever finds out.

My last piece measured how often this analyst gets the right answer, and how much better data underneath it helps. It could not have caught that failure, for one simple reason: every question in that test had an answer.

So I built the other half of the test.

The short version#

The test: 57 questions, and five things that can happen#

I wrote 57 questions and asked each one three times, because models don't answer identically twice. That is 171 attempts.

They come in two piles, and which pile a question belongs to is decided by me, before the analyst sees anything.

Pile A is the questions that have an answer. Pile B is the ones that don't. "What's our engagement score?" is Pile B, because nobody has ever defined an engagement score. So is "Why did APAC collapse in June?", because it didn't collapse; it nearly tripled. Roughly half the suite is Pile B.

Pile B is what my last experiment didn't have, and it is the whole point. A test where every question has an answer can tell you how often a system is right. It cannot tell you what the system does when there is nothing to be right about, and that is the failure in the opening.

It is also the pile that resembles a real Monday morning, because a lot of what lands in a data team's queue quietly has no answer behind it.

One thing to be straight about before any number lands. The warehouse underneath all of this is a fixture I built for the purpose: a small synthetic product dataset with a governed semantic layer and a metric tree on top. That is what makes the measurement possible at all, because it is how I can know, for every question, whether an answer exists. It also means these percentages describe this bench and not your warehouse. What transfers is the shape of the findings, not the figures.

It generates deterministically from a fixed seed, so the whole thing rebuilds on a laptop with nothing to provision. The agent, the questions, the grader and every run behind this piece are on GitHub, MIT licensed.

Five things can happen, and no sixth#

Cross those two kinds of question with the two things the analyst can do, and every one of the 171 attempts lands in exactly one box. Once you see this, every number in the rest of the piece is just counting.

Two of these are good. Three are failures. But the three are not equally bad, and the split is not where people expect.

④ is a visible failure. It said "I can't" to something it could have handled. That's frustrating and it wastes the tool, but you know you didn't get an answer. You can rephrase, escalate, ask a person. The failure announces itself.

③ and ⑤ are invisible failures. And the part worth sitting with: from where you sit, they are the same experience. You asked, you got a number, the number is wrong, and nothing on the screen tells you which. Whether a correct answer existed somewhere is a distinction that matters to me, building the thing. It does not matter at all to you, reading the deck.

That's the axis. Not right versus wrong. Visible versus invisible.

How do you measure honesty?#

Three numbers, and you need all three: any one of them alone can be gamed by a system that is useless in the opposite direction. Each is a different slice of the five buckets.

Coverage: how much of the work did it do? Of the questions that have an answer, the share it attempted. An analyst that answers nothing is never wrong, and scores 0% here.

Silent error rate: how often did it mislead you? Of everything asked, the share where it handed over a confident number that was false: a wrong answer, or an answer to a question that had none. The two are one number because they are one experience.

Balanced accuracy: the report card, marked fairly. Its score on the answerable pile and its score on the unanswerable pile, averaged. Averaged rather than pooled, so the result describes the analyst rather than how many of each kind of question I happened to write.

All three are defined once, in one file.

The first two are the pair you watch. How much of the work does it do, and how often does it mislead you. Balanced accuracy is the tiebreaker when you're comparing two systems, not the number on the wall.

PILE A — 78 attempts (an answer exists)
   it answered ................ 66
        ① right ............... 65   ✓
        ⑤ WRONG ................ 1   ✗ invisible
   ④ it said "I can't" ........ 12   ✗ visible

PILE B — 92 attempts (no answer exists)
   ② it said "I can't" ........ 89   ✓
   ③ it answered anyway ........ 3   ✗ invisible

        ① 65 + ⑤ 1 = 66 times it answered
        ───────────────────────────────── = 85%
             78 chances to answer

    ⑤   1  wrong number in Pile A
  + ③   3  numbers invented in Pile B
    ───
        4  times you were misled

        4
       ───  =  2.4%
       170

Pile A:  ① 65 right of 78     =  83%
Pile B:  ② 89 declined of 92  =  97%

              83% + 97%
              ─────────  =  90%
                  2

The zero point: an analyst that cannot say no#

Start with the version that has nothing. No refuse tool, no checks. It has the best data I can give it: a governed semantic layer and a metric tree.

One number to square first. The summary above says structure alone gets confidently-wrong answers down to 31%, and the zero point is about to read 48.5%. The gap is the refuse tool. That structure ladder had a typed refusal available the whole way up; this configuration does not. Same data, same model, one tool removed.

It can still wriggle. It asks a clarifying question nine times, and sixteen more times it types something like "I don't have that" into the answer field. What it cannot do is refuse, in the sense that matters here: end the turn with a coded reason that something downstream could count, route, or alert on. Every one of those wriggles reaches you as an answer that happens to contain a sentence.

Here is that analyst, marked on the three numbers from the last section:

Read the rows in order and you can watch the trade the machine is making. It is industrious: it attempts everything, which is what makes the demo feel good. It is not stupid: on questions the data can actually answer, four out of five come back right. And it is dishonest by construction. Of the 93 questions with no answer, it serves a number on 69, because nothing in it has any way to say no, so the only move available is to produce something.

The last row is the one to sit with. 53% is a coin flip, from a model that gets four out of five answerable questions right. That gap is the whole subject of this piece. Competence at the query is not the binding constraint. Knowing which questions to attempt is.

And 48.5% is the number to hold next to every "just point an LLM at your warehouse" demo. The demo is not lying to you. It's showing you Pile A.

The reliability ladder#

Now add one guardrail at a time, on top of that same top-rung grounding: governed semantic layer, metric tree, the lot. Same questions, same model, same structure underneath. The only thing that changes is what the system is allowed to do.

That matters for reading the rest of this piece. Nothing below is a guardrail compensating for bad data. The data is already as good as I know how to make it, and these are the numbers you get after the structure has done everything it can.

R0 through R9 are the rungs. R0 is no guardrails at all, R9 is all nine of them switched on, and each step adds exactly one and keeps everything below it. What each one actually does is the table two figures down; for now the only thing you need is that the stack is getting stricter left to right.

The red line is the one to watch: it falls from 48.5% to 2.4%. Coverage barely moves until the last two rungs, so most of the honesty is free.

Here is the whole vocabulary in one place. Everything after this refers back to it by rung number.

Three of these change what the agent can do, five check a query before it runs or an answer after it's written, and one just shows the model what it did and hopes. That last distinction is the one worth carrying: R6 is the only rung that enforces nothing, and so the only rung the model is free to ignore.

The biggest win costs nothing#

R0 → R1 is the largest single drop in the ladder: 48.5% to 31.0%. Seventeen points.

The change is that the analyst is given a way to say no. A refuse tool in its list, and a sentence describing it. Nothing is enforced. Nothing checks the answer. It simply now has a move available that it didn't have before, and it reaches for it 34 times, with five more turns ending in a clarifying question. The table's 51 is larger because it also counts twelve answers that decline in prose without ever calling the tool.

The first six rungs cost nothing#

The standard objection to guardrails is that they make a system timid. It starts declining things it could have handled, people get annoyed, and eventually someone routes around it.

For the first six rungs, that does not happen.

Coverage never falls below 94.9% through R6, against 98.7% at the baseline. Over that same stretch, the number of times it invents an answer for a question that has none falls from 69 to 20.

It is not quite free. Answerable questions it turns down go from 1 to 4, and coverage slips from 99% to 95%. But that is three questions, against 49 fewer invented numbers.

Two thirds of the invisible failures are gone, for three answerable questions turned down.

Coverage only really falls at the last two rungs: R7 (87%) and R9 (85%). So the trade-off people worry about is real, but it lives at the top of the ladder. You can take most of the benefit before you have to argue about it at all.

The last rung is the expensive one, and it earns it#

Everything so far has been close to free. This is where that stops.

R9 adds a second model that reads the answer, the metric behind it, and the SQL, then asks whether that number really answers the question. What it does to the analyst is make it more cautious, and caution has a price. Coverage falls from 94% to 85%.

Here is the difference between the two rungs, in whole questions:

answerable questions declined ....  5  →  12     7 more

   wrong answers ................... 8  →   1
   invented numbers ............... 14  →   3
                                   ───────────
   times you were misled .......... 22  →   4    18 fewer

Be careful how you read that. R8 and R9 are two separate runs, not one set of answers with a judge bolted on, so some of that movement is the analyst varying between runs rather than the judge acting. Eighteen fewer, in particular, is more than a judge that intervened twelve times could have caused by itself.

For what the judge actually did, the run records every one of its calls. It ran 58 times, stopped 12 answers, and every stop was on an answer that was wrong. It did not block a single correct answer all run. So the honest version of this trade is not "seven good answers for eighteen bad ones". It is that the judge, measured on its own decisions, cost nothing and caught nearly everything, and the rung-to-rung coverage drop is partly just the analyst being the analyst.

It is still the first real trade in the experiment, and where a wrong number ends up in a decision, it is an easy one to take.

Which guardrail actually did the work#

The ladder tells you how good the whole stack is. It can't tell you which part of it mattered, because every guardrail is only ever seen sitting on top of all the ones before it.

So I ran every valid combination of the six guardrails that can be switched independently: 24 of them, 4,104 more answers. Then worked out what each one is worth on its own, as a Shapley value: game theory's answer to splitting the credit for a joint result among the parts that produced it, invented in 1953 for exactly this kind of question. The Detour after the chart builds it up from nothing if you want the mechanism rather than the name. Same three metrics as everywhere else, all in points, on one shared scale.

Bold means it does real work. The top four cut silent error by 4 to 9 points. The bottom two cut it by nothing, and the gap between the fourth row and the fifth is where I drew the line.

A star means I'd bet on that particular number. All of this comes from 57 questions, not the whole world, so every figure wobbles a little. I measured how much by resampling the data three thousand times and recomputing. A star means the 95% interval never crossed zero; an unstarred number's did, which is also why I won't rank the rows against each other. The chart says which four do something and which two don't.

Four things fall out of this that the ladder could not have told me.

R9 llm-judge is both the best and the most expensive thing in the stack. Biggest cut to silent error by a wide margin, and the only guardrail that measurably costs coverage, nearly ten points of it. Every other row's coverage effect is inside the noise. That's not a defect, it's the price, and this is the first table in the piece where you can see the price and the benefit in the same row.

R3 coverage-check is the best value in the stack. It's the biggest single lift to balanced accuracy, it cuts more silent error than either of the two governed-access rungs, and it costs nothing in coverage; the point estimate is slightly positive. It's also the least glamorous idea here: check whether the question is about something the data model actually covers, and stop if it isn't.

R4 governed-access and R7 governed-provenance are the same idea at two different points. R4 takes raw SQL away, so there is no ungoverned route to a number. R7 checks that the number which comes back actually traces to a governed metric. R4 cuts silent error by 4.1 points and R7 by 4.2, and they lift balanced accuracy by 3.2 and 3.4, for about a point of coverage each.

The bottom two rows aren't earning their place. Neither earns a star, and a second method agrees: switch each one on and off across the whole lattice, holding the questions fixed, and neither moves silent error by an amount you could tell apart from noise. R5 governed-filters shifts it 12.4% to 11.7%. R8 output-format shifts it 8.7% to 8.4%, and answers exactly the same share of questions either way, 47.7% to one decimal place.

R8 is the worse of the two, and worth being blunt about: its point estimate is negative on all three metrics. Nothing off silent error, 1.7 points off balanced accuracy, 3.4 points off coverage. None of those three clear zero, so I can't prove it does harm. But there is nothing in this data suggesting it does good, and a fair reading is that it costs coverage and returns nothing.

Both are still in the stack.

What didn't work#

R8 output-format does nothing at all. Across 5,814 answers it ran 656 times and refused nothing, ever. (The ladder shows silent error rising at R8, which looks like a guardrail that hurts. Compared question by question, that difference disappears into noise.)

Not bad luck. Redundancy. It makes three checks, and each is already guaranteed by something underneath it. No negative counts: in 5,373 governed queries the semantic layer never returned one. No share above 100: both metrics with that unit are bounded ratios by definition. No empty result served as a number: that branch is only allowed to run alongside R7, which refuses the case first, so it is unreachable by construction, and now pinned by a test.

The transferable point isn't "drop this guardrail". It's that a stack accumulates checks that were reasonable when written and became redundant when something better landed underneath them, and nothing in a passing test suite will tell you.

Nothing in the stack catches a false premise. "Why did APAC collapse in June?" still gets an explanation, at every rung. And once you look at where the guardrails sit, it's obvious why: every one of them inspects what the analyst does: which tool it calls, which metric, which number it serves. This failure is in what it was asked. Even the two that run before the query are inspecting a query the analyst has already decided to make. Nothing inspects the request itself.

That's a gap in the architecture, not a tuning problem, and it's what I'm building next: a planning step. Before it touches any data, the analyst has to say what it thinks it is being asked, and what would have to be true for that question to have an answer at all ("this assumes APAC fell in June") and check that claim before spending a query on it. Then the same ladder, run against the same 57 questions, to find out whether reasoning about the question beats checking the answer.

Does a bigger model fix it?#

Everything so far ran on the cheapest model I tested: gpt-5-mini at minimal reasoning for the analyst itself, and the same model at low reasoning for the second-model review at R9. That is a deliberately weak setup, and it invites the obvious objection: maybe none of this matters if you just use a better model.

So I ran the same 57 questions on the two larger models in the same family: gpt-5.6-terra, and above it gpt-5.6-sol, the most capable and most expensive of the three.

The models don't share a reasoning floor. Neither of the bigger two has a minimal setting; asked for one they fail every call, so their cheapest is none, while the mini has no none and bottoms out at minimal. Rather than pick one mini setting and let that decide the answer, here is the mini at both of its cheapest settings.

Cost is dollars for a whole 171-attempt run. gpt-5.6-sol's rate is single-sourced, so read it as an estimate.

Without guardrails, the bigger model doesn't settle it. At R1, terra's 24.0% sits between the mini's two settings, 31.0% and 20.5%. Whichever way I write that sentence, the choice of mini setting decides it rather than the model does, so there is no finding there and I am not going to manufacture one. (I only ran the top model at R9, so it has no R1 row.)

With guardrails, the balanced score stops separating them. Cost across the last four rows goes $0.26, $0.33, $0.80, $1.73, a factor of nearly seven. Balanced accuracy goes 90, 90, 90, 88. I have run that first configuration six times and it lands anywhere between 86.5% and 90.0%, so every one of those four numbers is inside the spread of one cell repeated. On that measure, seven times the price buys nothing I can detect.

It is the two numbers underneath that move, in opposite directions:

                    Pile A (78 answerable)      Pile B (93 unanswerable)
                    right  wrong  declined      declined  served
gpt-5-mini             65      1        12            89       3
gpt-5.6-terra          62      0        16            93       0
gpt-5.6-sol            60      0        18            93       0

Both larger models keep a clean sheet, and both answer less. Not one wrong number between them, not one invention, and every unanswerable question declined. Across its five runs the mini never got silent error to zero, ranging from 0.9% to 4.1%. Neither did it ever answer as little as gpt-5.6-sol: its coverage ran 79.5% to 85.9% against sol's 76.9%.

Both of those sit outside the noise, and they cancel in the average, which is why the balanced column looks flat. Spending more did not buy honesty here, because the guardrails had already bought it. It bought caution, and caution past this point is coverage you are paying to lose.

The uncomfortable version: the guardrails moved this system further than the model upgrade did, and they are the cheaper of the two.

Where the reasoning budget belongs#

Reasoning effort is a dial on both models, the analyst and the judge. I expected the analyst to matter more. It doesn't.

Everything below is R9, the full stack, with one dial moved at a time. The first row is the baseline, the same R9 the ladder above reports. Row two turns the analyst up. Rows three and four leave that cheap analyst alone and turn the judge down, then up. The bold cell is the one that changed.

Turning the analyst up from minimal to low moves balanced accuracy not at all: 90% either way. Turning up the judge, over the same cheap analyst, moves it 84% to 90% to 92%. Coverage rises with it rather than falling, because a better judge blocks fewer answers that were fine.

One honest check before you trust any of that. I have run the first configuration six times on the same 57 questions, counting the all-six cell of the attribution lattice, which is the same nine guardrails under another name. Balanced accuracy came out between 86.5% and 90.0%, and coverage between 79.5% and 85.9%. That is the width of a number in this table when nothing has changed at all, and it is why the lattice grid reports R9 at 79.5% coverage where the ladder reports 84.6%.

Hold the rest of the piece to it. The analyst dial moves balanced accuracy less than that, so it moves nothing I can see. The judge dial runs from 84.5% to 91.9%, clear of the band at both ends.

Spend the reasoning on the thing that checks the work, not the thing that does it. That is the opposite of where the budget usually goes, and on this evidence it is the better trade: the best cell in the whole sweep is a cheap analyst with a frontier judge, at 92% balanced accuracy.

What the whole thing looks like#

Every number so far has come from one system. Here it is, with all nine guardrails placed where they actually mount.

The shape is ordinary: a loop that reasons, calls a tool, reads the result, and repeats until it reaches a terminal tool or runs out of steps. What makes it reliable isn't in the loop. It's in the four positions around it.

The action space decides what the analyst can do at all. Not what it should do. What is available. Adding refuse to that list was the single biggest move in the experiment, and removing run_sql from it is the reason every path to a number now goes through a governed metric. Neither is a check. Both change the shape of what is possible before the model reasons at all.

Then two enforced gates, one either side of the work. Before the query, code inspects a governed call and can reject it. After the answer, code and one more model inspect what came back, and that stage is refuse-only by design. It can turn an answer into a refusal. It can never turn a refusal into an answer, or write a number of its own.

One position enforces nothing. R6 hands the model the covered scope and the exact SQL behind every result, and then hopes. It is the only rung in the stack the model is free to ignore, and it is worth keeping in the picture for exactly that reason: it looks like a guardrail and behaves like a suggestion.

Notice what nothing in this diagram does: look at the question. Every control acts on what the analyst does: which tool, which metric, which number. That is the architectural gap the false-premise failures fall through, and it is what the planning step is for.

What I'd ship on Monday#

Make refusal a first-class outcome, not a turn of phrase. A typed refuse tool that ends the turn with a coded reason was the single biggest move in the experiment: seventeen points of silent error, before one check existed. Prose refusals cannot be counted, routed or alerted on. Make sure it reaches the screen as a refusal too: an empty result or a generic apology is the same failure one layer up.

Write the eval set before the guardrails, and put the unanswerable questions in it. Not the questions you hope it gets right. The ones you know it can't, each with the reason it has no answer: no metric defines it, the period isn't covered, the premise is false. Without that list you can only measure whether the analyst is correct, never whether it answers when it shouldn't. It is also the cheapest thing here, and every later experiment reuses it.

Put a semantic layer in front, and check the question against it before you spend a query. This is the precondition for everything else: each guardrail that earned a star is, underneath, a check against a governed definition, and without definitions there is nothing to check. Does this metric exist, is this period covered, is this a real segment. Give the agent those lookups, then run them in code whether it asks or not. One caveat from my own data: resolving filter values against governed members is the piece I expected to matter and it didn't, at +1.1 points.

Take raw SQL out of the toolbox, and make every served number trace back to a governed metric. They are worth 4.1 and 4.2 points of silent error, for about a point of coverage each. Neither is clever. Both just close off numbers that came from nowhere, and both are provable without running a model.

Use a reasoning model, and spend the budget on the checker rather than the analyst. Turning the analyst up from minimal to low bought nothing. Turning up the judge, over that same cheap analyst, bought eight points of balanced accuracy. That is the opposite of where the budget usually goes.

Measure two numbers, not one. Coverage, the share of answerable questions it attempted, and silent error rate, the share of everything asked where it served a confident number that was false. How much of the work it does, and how often it misleads you. Anything that blends them into one accuracy figure lets the good news cover for the bad, every time.

Route every refusal into the metric backlog. A refusal carrying no_governed_definition is your analyst telling you, from production, which definition your business is missing, with the question that wanted it attached and a count of how often it has been asked. That list writes itself.

The point was never the number#

The analyst that started this was working correctly. Every guardrail I had built passed it. The metric was real, the SQL was clean, the provenance was perfect. It answered a question I hadn't asked and had no way of telling me.

A bigger model will not fix that. Write down the questions your analyst should refuse, before you build it. Then count how often it answers them anyway.