Agentic Analytics: How Much Does Grounding Actually Buy You?

The pitch for an AI analyst is irresistible: point a model at your warehouse, ask a question in plain English, get an answer back without waiting on the data team. Every vendor says it works, and the benchmark numbers they publish are high.

I wanted my own numbers. Not from a staged demo, and not from a vendor benchmark tuned to the vendor's own product, but from a small setup I controlled end to end, where I could see exactly what the model was standing on and change one thing at a time. How good does an AI analyst actually get, and how much of it comes down to the grounding underneath?

What makes that hard to answer honestly is that a wrong answer doesn't look wrong. Ask a model for your activation rate and it may hand you 76.9% in one clean sentence, no hedging, when the real number is 53.2%. It invented the definition, because nobody told it that "activated" here means reaching first value within seven days. It doesn't crash or refuse. It gives you a board number that happens to be wrong, and nobody in the room can see it.

I argued in Data Modelling in 2026 that the fix for this isn't a smarter model, it's the structure underneath it. I've also lived it with no model in the room. At a fintech where I ran data, our deposit success rate, the number the business steered by, read too good for months. It was assembled by hand: part from the warehouse, part from a CSV the payment provider emailed us, glued together in a spreadsheet full of magic rules, and nobody had ever agreed what a "successful deposit" even was. Writing that definition down, one governed rule wired straight into the provider's data instead of the manual patchwork, is what fixed it. AI doesn't invent this failure; it just runs it faster, with no spreadsheet to blame and nobody to catch the guess.

Arguing it and living it still aren't proof, so I built the smallest honest test I could and measured it.

The mental model: a reliability stack

Give a capable model your raw database and you have a sharp analyst on their first day, with nobody to ask. It writes fluent SQL. It also has no idea that "active users" is supposed to exclude staff, that the region which launched in May has test data sitting in front of it, or that annual plans are billed as one yearly lump. That missing knowledge is the analyst's job, and a data team supplies it, in layers.

Each layer is a kind of grounding, and they are not equal. What separates them is how firmly each one holds a rule the model would otherwise have to guess:

Read it bottom to top. The semantic layer is the first rung that is deterministic: the definition of "active users" is the metric, so once the model calls it the count is right by construction. It can still reach for the wrong metric or slice it wrong (a measure like MRR won't simply sum across time), but the definition itself is no longer something it can get wrong. Verified examples are approved question-to-query pairs it copies. A knowledge base is a written rule it may or may not apply. A metric tree is different in kind, not a lookup but a way to reason about why a number moved. Reliability falls as you climb, until the top rung buys something none of the others can.

The test

The experiment is one move: ask the same 25 business questions six times, and change only the structure under the model. Model frozen, questions frozen, so anything that moves is the structure talking, not prompt luck. Each rung keeps everything below it and adds one thing, so the only difference between two neighbours is what the higher rung adds. It's a public repo, a few hundred lines of Python and a DuckDB file that runs on a laptop.

The data is a synthetic habit-tracking app, built messy on purpose, with the traps a real warehouse has: staff accounts sitting in production, an events table that mixes app opens with habit completions, revenue that needs unbundling. It also carries one planted anomaly, where value moments fell last week because a notification change cut reminders, so "why did it drop" has a real answer to grade against.

What the climb looks like

Here's a frontier model, one of today's GPT-5.6 generation, on that ladder. Same 25 questions every time; the only thing that changes is the rung it stands on.

% of 25 questions correct · mean of 5 runs

41%±5

46%±4

66%±4

81%±2

79%±2

92%±3

1Messy data

2Star schema

3Semantic layerbiggest jump

4+ Verified examples

5+ Knowledge baseno lift

6+ Metric treediagnosis

That 92% isn't a promise your warehouse will hit it. The bench is a toy on purpose: a real warehouse would bury the signal under a hundred confounders, and this one is small enough to read in a sitting. Think the traps are rigged to flatter the layers? Change one, re-run, and see if the shape holds. A bench like this shows the direction and the mechanism; whether it holds at real scale is the job of the outside benchmarks I lean on below.

The average also hides the more useful pattern. Each kind of question has a rung where it switches from wrong to right, and the harder the question, the more structure it takes:

• Simple facts ("value moments in June") work almost from the start.
• Definitional questions ("how many active users") need the semantic layer.
• Business-knowledge questions ("real acquisition spend, minus the test channel") need verified examples.
• "Why did it move" questions need the metric tree at the very top.

Four things are worth pulling out of that climb, and two of them surprised me.

The semantic layer is where it becomes an analyst

The biggest jump is rung 2 to rung 3, and it's the moment the model stops guessing definitions. A semantic layer is a set of governed metrics: one agreed way to compute "active users," a "power user," activation, MRR, so the number is right by construction. None of these are things a model can infer from data, however clean. That active users exclude staff, that activation means first value within seven days, that an annual plan enters MRR as a twelfth of its price each month, all of it has to be written down somewhere the model is forced to go through.

The name matters less than the mechanism. What clears this rung is a governed definition, encoded rather than invented, and you can build that as a dedicated semantic layer, a dbt model, or even a plain view per metric on the star schema. A view that already excludes staff and applies the window hands the agent the same right number a metric would. A dedicated layer pays for itself later, when the questions turn open-ended and you want to slice one definition many ways without hand-building a view for each.

That is why a clean schema alone barely moved the score. The model writes good SQL against tidy tables and still counts the wrong thing. Ask for active users last week and it answers 919, counting everyone; the company's own definition, minus staff and test accounts, says 886. The number comes right only once the definition is encoded, not inferred.

This isn't just my toy. dbt's 2026 benchmark measured the same climb and named the stakes well: "With text-to-SQL, failure looks like a plausible but incorrect answer. With the Semantic Layer, failure looks like an error message" (Ganz and Perigaud, dbt). And the frontier models that clear 90% on tidy textbook databases solved about a fifth of the tasks on real enterprise ones when Spider 2.0 launched, a gap that has narrowed since but not closed; a separate benchmark saw LLM accuracy on enterprise questions climb from 16% to 54% once the raw schema was replaced with a curated knowledge graph (Sequeda et al.). The gap is the definitions.

One design choice sits under all of this: how the agent is allowed to reach the layer. Mine can still fall back to raw SQL at every rung (deliberately, so I could see what it reached for), but in production you take that away, so it can only ask for governed metrics by name, never an ungoverned query. That constraint is a tool-design choice, and MCP has become the standard way to expose it: the agent reads a catalogue of governed metrics and asks for them by name instead of writing SQL.

Verified examples deliver knowledge; a prose knowledge base doesn't

This is the result I didn't expect, and the one I'd take to any team about to spend six months writing an internal "AI context doc."

Some knowledge no metric definition can hold: that the region which launched in May has a pre-launch test cohort to exclude, that the partnerships channel is an internal test and not real acquisition, that "retention" from a founder usually means frequency. This is the "give the AI business context" step, and how you deliver it decides whether it holds.

Delivered as verified examples, it holds. These are approved question-to-query pairs, each a scoped call into the governed metrics, the way Snowflake's Verified Query Repository works, and the idea agentic layers like Cube are built around. One example teaches the partnerships exclusion, and the related questions lock onto the right answer. The business-knowledge questions went from nearly all wrong to all right.

Delivered as a free-text knowledge base, the same rules stacked on top of the examples, it doesn't. Over five runs the knowledge-base rung never once improved accuracy for either model. It cost each about two points on average, the only rung that never earns its place. The reason is simple. An example is scoped and applied by matching, so it behaves the same way every time. A prose rule is applied whenever the model decides it's relevant, and that judgment varies: sometimes it helps, sometimes it adds noise, sometimes it takes a rule meant for one question and sprays it onto another.

The careful vendors already build in that order. Databricks ranks its own grounding in its docs: SQL expressions first, example queries second, free-text instructions "only as a last resort."

The metric tree is the frontier

The one thing none of the lower rungs can reliably do is answer "why did value moments drop." A metric tree is built for exactly that. It's a map of how the North Star breaks into its parts (how many users, how often, how deeply) and the empirical links under them, like reminders driving frequency, plus an engine that decomposes a change through the map and reports the cause. It's the difference between a model that writes SQL and one that can debug a business.

The tree has two kinds of edge, though, and they don't earn equal trust. The identity edges are pure arithmetic (value moments are users times frequency times depth), so decomposing along them is deterministic and safe, and honestly this part is just the semantic layer carried one step further. The influence edges, like reminders driving frequency, are causal guesses a person drew by hand. They're where the confident-wrong problem from the opening creeps back in, one level up: the tree can hand you a clean, plausible cause that happens to be the wrong one. The arithmetic is safe. The causation isn't.

Without the tree, the model brute-forces diagnosis: it pulls one metric after another for two periods and reasons by hand, and mostly it misses. With the tree, the flagship's diagnosis jumps from roughly one right answer in eight to seven in ten, and it gets there in a fraction of the work: a diagnostic answer that took about a dozen tool calls at the semantic layer takes three with the tree, because the decomposition runs in a single call rather than a pile of hand-built queries. This is the part most analytics stacks still don't ship. A semantic layer defines metrics on their own islands; the tree is the map of how they drive each other, and it's the natural home for the metric trees I keep telling teams to draw before they instrument anything.

One honest limit on that jump: the anomaly I planted (a notification change cut reminders, so frequency fell) is a path the tree already encodes as an influence edge. So the result measures whether the tree lets the model retrieve and confirm a cause a human already drew, not whether it can discover one nobody modelled. A tree is only as good as its edges; hand it a driver it doesn't encode and diagnosis is back to guessing.

Structure beats model size, until it doesn't

Does a bigger model help as much as better structure? I ran the same ladder on a cheap mini next to the flagship. The short answer: a cheap model plus structure is a real analyst for "what happened," but not for "why."

On raw data the mini is dangerous: it reported MRR more than four times too high. Give it a semantic layer and verified examples and it nearly catches the flagship, at a fifth of the cost. The metric tree lifts both, but not equally: with it the flagship gets most of its diagnoses right while the mini manages barely a third. It can read a governed number and take a decent stab at why it moved; it just can't reason about the why as reliably as the bigger model.

That points at a cost structure a real team can run: reporting on the cheap model, diagnosis on the flagship.

The cost of a right answer

Structure has a second dividend, the one a CFO notices: it lowers the cost of a right answer.

cost per correct answer · × = cheaper than raw data

1×$0.0308

1.8×$0.0168

2.8×$0.0112

4.1×$0.0075

3.3×$0.0094

3.5×$0.0089

1Messy data

2Star schema

3Semantic layer

4+ Verified examples

5+ Knowledge baseticks up

6+ Metric tree

Across the ladder that cost falls roughly 4×, from about $0.031 to a floor near $0.008 at the verified-examples rung. The mechanism isn't the intuitive one, though. Cost per answer does fall at first, as the flailing stops, but it bottoms early and then climbs back over the top two rungs, because the richer grounding the model reads outweighs the fewer queries it runs. What keeps falling is cost per correct answer, and it falls because the hit rate climbs: the dollars you save are the ones you were spending on wrong answers.

The metric tree is the clearest case: it pushes accuracy higher without lowering the cost of a right answer any further, because its cheaper querying and its heavier grounding roughly cancel. Extra correctness at the same unit price. And the one rung where the cost of a right answer ticks back up is, predictably, the knowledge base.

The point was never the model

Six rungs, and the shape of the answer is the one I keep coming back to. The model is the engine. Structure is the map, and every rung of it is something a data team builds, not something you buy: a deterministic semantic layer at the bottom, fragile free-text near the top, and the metric tree, the map of how the business actually moves, as the capstone.

The work, then, is unglamorous. You push knowledge down that ladder, out of people's heads and brittle prose, into governed structure a machine can be trusted to act on: a semantic layer so each number comes out right, a metric tree so the model knows which number to ask about. That structure is the spine the analyst leans on. Get it right, and the confident wrong answer stops reaching the room.