The AI Analyst Needs a Spine

I asked a cheap AI model a question any founder asks: what's our activation rate?

It answered 76.9%. One clean sentence of justification, no hedging. The real number was 53.2%. It hadn't miscounted anything. It had invented the definition. "Activated" at this company means reaching first value within seven days of signing up. The model had no way to know the window was seven days, so it chose its own, and handed me a number 24 points too high, for a figure that goes in a board deck.

You would not have caught it. 76.9% is plausible, it holds steady across re-runs, and it arrives without a flicker of doubt. That is the failure worth worrying about when you put AI on top of your data. The model does not crash or refuse. It hands you a clean, confident number that happens to be wrong, and nobody in the room can see it.

A few months ago I argued in Data Modelling in 2026 that "AI-ready data" is a direction, not a checkbox: a clean schema buys you reliable simple answers and nothing more, and the hard questions need a system built around the model. It's easy to assert, so I built the smallest honest test I could and measured it.

The mental model: a reliability stack

Give a capable model your raw database and you have a sharp analyst on their first day, with nobody to ask. It writes fluent SQL. It also has no idea that "active users" is supposed to exclude staff, that the region which launched in May has test data sitting in front of it, or that annual plans are billed as one yearly lump. That missing knowledge is the analyst's job, and a data team supplies it, in layers.

The layers are not equal, and that is the whole point. What separates them is how firmly each one holds a rule the model would otherwise have to guess:

Read it bottom to top. The semantic layer is the first rung that is deterministic: the definition of "active users" is the metric, so the model calls it and cannot get it wrong. Verified examples are approved question-to-query pairs it copies. A knowledge base is a written rule it may or may not apply. A metric tree is different in kind, not a lookup but a way to reason about why a number moved. Reliability falls as you climb, until the top rung buys something none of the others can.

The test

The experiment is one move: ask the same 25 business questions six times, and change only the structure under the model. Model frozen, questions frozen, so anything that moves is the structure talking, not prompt luck. It's a public repo, a few hundred lines of Python and a DuckDB file that runs on a laptop.

The data is a synthetic habit-tracking app, built messy on purpose, with the traps a real warehouse has: staff accounts sitting in production, an events table that mixes app opens with habit completions, revenue that needs unbundling. It also carries one planted anomaly, where value moments fell last week because a notification change cut reminders, so "why did it drop" has a real answer to grade against.

The rungs accumulate: each one keeps everything below it and adds a single thing. Rung 4 is the semantic layer plus verified examples, not examples in place of it. That's what keeps the comparison clean, because the only difference between two neighbouring rungs is the one thing the higher rung adds.

What the climb looks like

Here's a frontier model, one of today's GPT-5.6 generation, on that ladder. Same 25 questions every time; the only thing that changes is the rung it stands on.

% of 25 questions answered correctly

36%

48%

68%

84%

80%

92%

1Messy data

2Star schema

3Semantic layerbiggest jump

4+ Verified examples

5+ Knowledge basefragile

6+ Metric treediagnosis

The average hides the more useful pattern. Each kind of question has a rung where it switches from wrong to right, and the harder the question, the more structure it takes:

• Simple facts ("value moments in June") work almost from the start.
• Definitional questions ("how many active users") need the semantic layer.
• Business-knowledge questions ("real acquisition spend, minus the test channel") need verified examples.
• "Why did it move" questions need the metric tree at the very top.

Four things are worth pulling out of that climb, and two of them surprised me.

The semantic layer is where it becomes an analyst

The biggest jump is rung 2 to rung 3, and it's the moment the model stops guessing definitions. A semantic layer is a set of governed metrics: one agreed way to compute "active users," a "power user," activation, MRR, so the number is right by construction. None of these are things a model can infer from data, however clean. That active users exclude staff, that activation means first value within seven days, that MRR is annual revenue over twelve, all of it has to be written down somewhere the model is forced to go through.

That is why a clean schema alone barely moved the score. The model writes good SQL against tidy tables and still counts the wrong thing. Ask for active users last week and it answers 919, counting everyone; the company's own definition, minus staff and test accounts, says 886. The number comes right only once the definition is encoded, not inferred.

This isn't just my toy. dbt's 2026 benchmark measured the same climb and named the stakes well: "With text-to-SQL, failure looks like a plausible but incorrect answer. With the Semantic Layer, failure looks like an error message" (Ganz and Perigaud, dbt). And the frontier models that clear 90% on tidy textbook databases solve only about a fifth of the tasks on real enterprise ones (Spider 2.0). The gap is the definitions.

Verified examples deliver knowledge; a prose knowledge base doesn't

This is the result I didn't expect, and the one I'd take to any team about to spend six months writing an internal "AI context doc."

Some knowledge no metric definition can hold: that the region which launched in May has a pre-launch test cohort to exclude, that the partnerships channel is an internal test and not real acquisition, that "retention" from a founder usually means frequency. This is the "give the AI business context" step, and how you deliver it decides whether it holds.

Delivered as verified examples, it holds. These are approved question-to-query pairs, each a scoped call into the governed metrics, the way Snowflake's Verified Query Repository or Cube's Certified Queries work. One example teaches the partnerships exclusion, and the related questions lock onto the right answer. The business-knowledge questions went from nearly all wrong to all right.

Delivered as a free-text knowledge base, the same rules stacked on top of the examples, it doesn't. Across the three models I ran, that rung changed accuracy by −4 points, 0, and +4: the only rung whose sign flipped from model to model. The reason is simple. An example is scoped and applied by matching, so it behaves the same way every time. A prose rule is applied whenever the model decides it's relevant, and that judgment varies: sometimes it helps, sometimes it adds noise, sometimes it takes a rule meant for one question and sprays it onto another.

The careful vendors already build in that order. Databricks ranks its own grounding in its docs: SQL expressions first, example queries second, free-text instructions "only as a last resort."

The metric tree is the frontier

The one thing none of the lower rungs can reliably do is answer "why did value moments drop." A metric tree is built for exactly that. It's a map of how the North Star breaks into its parts (how many users, how often, how deeply) and the empirical links under them, like reminders driving frequency, plus an engine that decomposes a change through the map and reports the cause. It's the difference between a model that writes SQL and one that can debug a business.

Without the tree, the model brute-forces diagnosis: it pulls one metric after another for two periods and reasons by hand, and mostly it misses. With the tree, the flagship goes from one correct diagnosis in five to three. And on the model I traced call by call, a diagnostic answer that took a dozen-plus tool calls at the semantic layer took a handful with the tree, because the decomposition runs in a single call rather than a pile of hand-built queries. This is the part almost nobody has shipped. A semantic layer defines metrics on their own islands; the tree is the map of how they drive each other, and it's the natural home for the metric trees I keep telling teams to draw before they instrument anything.

Structure beats model size, until it doesn't

Does a bigger model help as much as better structure? I ran the same ladder on a cheap mini next to the flagship. The short answer: a cheap model plus structure is a real analyst for "what happened," but not for "why."

On raw data the mini is dangerous: it reported MRR of $11,448 against a real $2,685. Give it a semantic layer and verified examples and it nearly catches the flagship, at a fifth of the cost. But the metric tree, which lifts the flagship, hurts the mini: it never solves a single diagnosis, and the extra tooling drags its other answers down. It can read a governed number. Ask it why the number moved and it's lost.

That points at a cost structure a real team can run: reporting on the cheap model, diagnosis on the flagship.

The point was never the model

Six rungs, and the shape of the answer is the one I keep coming back to. The model is the engine. Structure is the map, and every rung of it is something a data team builds, not something you buy: a deterministic semantic layer at the bottom, fragile free-text near the top, and the metric tree, the map of how the business actually moves, as the capstone. (The numbers are exact but each question runs once, so a cell wobbles a few points between runs; the patterns hold, and the repo runs the whole thing end to end.)

The work, then, is unglamorous. You push knowledge down that ladder, out of people's heads and brittle prose, into governed structure a machine can be trusted to act on. Get the number right and you've built a semantic layer. Know which number to ask about and you've built a tree. Do both, and the confident wrong answer stops reaching the room.

That structure is the spine an AI analyst leans on. Everything above it is just the model talking.