What It Takes to Trust an AI Analyst

An AI analyst can use governed data, execute valid SQL and still give someone a misleading answer.

I learned this after better structure moved accuracy from about 40% to 92% in the first experiment. The remaining failures became less obvious, not less important. The SQL ran. The number looked normal. But the system could still choose the wrong period, drop a segment, use an unsupported measure, or silently select one of two legitimate definitions.

That changed the question. I was no longer asking which model was best at analytics. I was asking what the system had to guarantee before an answer reached a user.

Across six experiments, the answer settled into four layers: a trusted data foundation, governed definitions, a bounded verification harness, and evaluation that can see silent failure.

Context makes an AI analyst capable. Verification makes the system reliable.

Context is necessary. It is not a control#

Two recent accounts of internal data-agent systems start with context. Anthropic describes data foundations, sources of truth, skills and validation. OpenAI describes table usage, human annotations, code-derived meaning, institutional knowledge, memory and runtime context. Both systems treat the model as part of a larger data product rather than an isolated text-to-SQL interface.

An agent needs this context. It cannot reliably infer a table's grain, a metric's scope or a company's internal vocabulary from column names alone. It needs curated definitions, lineage, examples, business context and access to live data.

But my experiments kept finding the same boundary. The presence of information did not guarantee that the answering path used it.

A free-text knowledge base made more business context available but did not improve accuracy on top of the structured semantic layer. In another test, the agent had two metric descriptions and could explain the difference when directly asked to compare them. When asked for the number, it chose one. A clarification tool and an explicit instruction to ask exposed the choice 0 times in 51 controlled attempts.

The model was not missing the information. The system was missing a control.

This distinction changes how an analytics agent should be designed. Context is what the model may use. A control is what the system checks regardless of whether the model remembers to use it.

Examples, descriptions and instructions belong in context. Applied filters, allowed dimension values, coverage windows, result values and required disclosures can be verified. Once a fact can be checked mechanically, leaving it to model compliance makes the system less reliable for no benefit.

Prompts and documents can be ignored. Facts become more reliable when they are encoded in the data model or a governed metric and checked before the answer is served. The repair experiment showed the consistent pattern: move important rules from optional context into the enforced workflow. That is what turns context into control.

A reliable AI analyst has four layers#

The four layers are not a technology stack or a vendor architecture. They are a division of responsibility. Each answers a different question, has a different owner, and catches failures the other layers cannot see.

These four responsibilities are familiar. Anthropic and OpenAI describe many of the same building blocks: trusted data, governed definitions and validation. My question was more practical: what can an ordinary team actually rely on?

The experiments added three lessons. The system, not the model, must expose competing definitions. It must distinguish questions it can answer, cannot answer or needs to clarify. And once a test set has influenced a fix, the next claim must be tested on fresh questions.

1. Trusted data foundation: can the data be trusted?#

This layer includes infrastructure, ingestion, the warehouse, foundational data models, clear documentation of what each row represents, lineage, freshness and data quality. These are familiar data-engineering responsibilities, and AI does not make them less important. It makes their hidden assumptions more expensive.

A human analyst may recognize that DE means Germany, know that one row represents an account-day, or remember that the current month is incomplete. An agent sees only what the system makes legible. If the grain, scope or coverage is absent, it fills the gap with a plausible interpretation.

The foundation owns facts about the data itself. It should make both the correct path and its boundaries explicit: what exists, what one row represents, how tables relate, when data is complete, and which transformations have been tested.

For a practitioner, the first test is simple: can an analyst reproduce the expected result directly from trusted warehouse data, at a known grain, without relying on private knowledge?

2. Governed definitions: what does the business concept mean?#

Governed definitions turn business language into executable logic. Trusted rows alone do not define revenue, retention or an active user. Each metric records its calculation, scope, owner, consumer and intended use.

A semantic layer such as Cube or dbt's MetricFlow already provides much of this: executable metrics, entities and governed joins. A prose note saying that revenue excludes refunds is useful context. An executable rule gives the system something it can run, compare and verify.

The closed-world ontology adds a boundary around what the system can support. It reuses the semantic model and lets the harness distinguish:

• An existing governed metric
• A calculation the data supports but the organisation has not defined
• A question the warehouse cannot support

It also records which owned definitions compete for the same business concept. The harness can then compare their results at runtime instead of leaving the relationship for the model to infer.

The goal is not the largest catalogue. It is the smallest set of definitions that covers real decisions without hiding absence or disagreement.

3. Harness: was the answer checked before serving?#

The harness is the bounded workflow around the model. It controls which tools are available, routes the agent toward governed computation, represents answer, clarification and refusal as explicit outcomes, records the trace, and runs verification before anything reaches the user.

This is where advice becomes enforcement. A prompt can ask the model to use the semantic layer. The harness can prevent an ungoverned calculation from silently replacing a governed result. A tool can show a competing metric. The harness can check that both values reached the final answer.

The model's first answer is a proposal. It becomes a served answer only after the workflow has checked the structural facts the system already knows how to verify.

This does not require turning the agent into a fixed script. The model can still explore, choose an analytical approach, ask useful questions and explain what the result means. The boundary is that it does not get the last word on facts that code can check.

Verification can fail too. If a check or a competing-definition query errors, the harness serves the governed value marked unverified rather than blocks, and records the failure as evidence.

4. Evaluation: can we see the failures users will not?#

Evaluation owns performance and reliability measurement: real business questions, expected results, question states, held-out suites, production evidence and the cost of each intervention.

This layer is independent of the model's explanation. A confident answer does not grade itself. A successful SQL query is not proof that the right question was answered. A refusal is not safe merely because no number was served.

Evaluation owns the definition of reliable behaviour. It must distinguish a correct answer from a plausible substitute, a necessary clarification from an unnecessary interruption, and a useful refusal from one that avoids an answerable question.

It must also be tested. In this work, broken answer keys, weak graders and provider rate limits all produced results that initially looked like agent behaviour. The measurement path is part of the product because it decides which failures the team can see.

The model should interpret. The system should verify#

The central design choice is not model versus code. It is deciding which kind of work belongs to each.

Language models are useful where meaning is contextual. They can infer what a user is trying to understand, connect business language to governed concepts, select an analytical approach, and explain the result in words appropriate to the audience.

Governed artifacts are useful where meaning must remain stable. Metric logic, grain, valid members, scope, owners and business definitions should not change because a prompt was phrased differently.

Deterministic checks are useful where the trace already contains enough evidence to decide. The system can verify that a named segment exists, that its filter reached the query, that a value matches the result, and that a required alternative appears in the served answer.

The allocation rule is straightforward:

Let the model interpret the question. Store durable meaning in governed artifacts. Verify every structural fact that the system can check.

Consider the word “customers.” The model may reasonably connect it to the real-account segment. The system can then verify that the segment exists, that its member values are valid, that the filter was applied, and that the served number matches the governed result. Only the first step requires language judgement.

This is also why highly prescriptive prompts are a poor substitute for architecture. A long prompt mixes semantic guidance with structural rules and asks the model to enforce both. The more robust design gives the model room to reason while moving checkable constraints into the path it cannot bypass.

Answerability should be checked, not guessed#

A metric catalogue tells the agent what has already been defined. It does not tell the agent whether an undefined request could be calculated from existing data or whether the required data is absent. Both situations produce the same catalogue result: no matching metric.

The harness therefore consults a closed-world ontology: a verified map of what the analytical warehouse captures, including its entities, measures, attributes, grains and approved relationships. For the purpose of serving an answer, that map is treated as complete. If a required input or join is absent, the system treats it as unavailable rather than asking the model to improvise.

The model interprets the request and identifies the required ingredients. The ontology verifies whether those ingredients exist and can be joined. That produces three outcomes: execute an existing governed metric, define and verify a calculation that the data can support, or refuse and name what is missing.

The closed world is a serving boundary, not a claim that the organisation knows everything. Missing concepts become explicit gaps that can be reviewed and added later. The companion article, How an AI Analyst Knows What It Can Answer, explains the mechanism and its limits.

That handles the case where no governed answer exists. The next problem is the opposite one: several governed definitions can answer the same question.

Governance does not always mean one metric#

Most definition conflicts are data debt. An old metric sits beside its replacement. Several names describe the same calculation. Two models count the same process at incompatible grains. These problems should be found offline and repaired.

Some disagreements are legitimate.

Product may define an active user as a real customer who used the product. Platform may include internal and test accounts because infrastructure load does not care who generated it. Both metrics can be owned, executable and necessary for a real decision.

In the experiment, the Product definition returned 886 and the Platform definition returned 919 for the same week. The difference was only 3.72%, small enough for either number to look normal in a board slide. Serving one without naming the other turned a governed result into a hidden choice.

The two definitions are one concept at one grain, a count over account-days, differing only by the internal and test-account filter:

metrics:
  - name: active_users     # Product · weekly product review · owner: product-analytics
    type: simple
    type_params: { measure: active_account_days }   # grain: one row per account-day
    filter: "{{ Dimension('account__is_internal') }} = false"   # excludes internal + test
  - name: active_accounts  # Platform · capacity planning · owner: platform-eng
    type: simple
    type_params: { measure: active_account_days }   # same grain, same measure
    # no filter: internal and test accounts included

preflight records the pair offline: the concept active_users_last_week, its members active_users and active_accounts, and the account__is_internal filter that separates them.

This requires two controls at different times.

First, an offline scan such as preflight finds definitions that may compete for the same business concept. It records the pair, their scopes, the filter that separates them, and the owner and consumer of each. Business owners decide whether the conflict is debt to remove or a legitimate pair to preserve.

Then the harness checks the pair at runtime. It executes both definitions with the user's exact period, dimensions and filters, against the same warehouse snapshot, so a difference in their values is a difference in definition and not a timing skew. If the values agree, the question continues normally. If they differ and the user has not already resolved the scope, the answer must ask or show both.

The contested-definitions experiment showed a clear progression. An instruction to clarify failed in every attempt, 0 out of 51. After the competing value was attached, the agent disclosed it in 50 out of 51 attempts. A final verification check caught the remaining miss and inserted the governed disclosure itself.

Legitimate disagreement can remain. Hidden choices cannot.

Evaluate three types of questions#

The answerability boundary and the ambiguity flow give the system explicit behaviour for zero, one or several valid answers. Evaluation should mirror those conditions.

An AI analyst faces three fundamentally different types of questions:

• Answerable: one governed answer fits the request.
• Unanswerable: the system does not have the data, definition or coverage required to answer.
• Ambiguous: several governed definitions fit, and the user has not selected one.

A reliable evaluation needs all three. Testing only answerable questions measures whether the agent can produce the expected result. Adding unanswerable questions also measures whether it refuses when no valid answer exists. Ambiguous questions test whether it exposes a choice instead of silently making one for the user.

This changes the meaning of correctness. For an ambiguous question, asking the user to choose can be correct. Showing every relevant governed value can also be correct. Serving one valid value as the only answer is a silent error. Refusing is unnecessary because the system can answer.

I reduce the outcome matrix to three rollout measures:

• Silent-error rate: how often the system serves a misleading figure without signalling a problem.
• Coverage: how often it serves a figure when a valid answer exists.
• Balanced accuracy: the average correct-handling rate across the question states represented in the suite.

These measures stay separate because they do not move together. Refusing more can lower silent error while destroying coverage. Answering everything can increase coverage while hiding more risk. Balanced accuracy prevents either extreme from looking healthy merely because the suite contains more of one question type.

Grounded-answer rate sits beside them as an assurance signal. It asks whether the claims in a served answer trace to governed results through the evidence graph. This can be measured offline and monitored in production, but it should not be averaged into correctness. A fully grounded answer can still be grounded to the wrong metric.

Groundedness tells us whether an answer can be inspected. It does not tell us whether the right answer was selected.

Fresh questions are the real test#

Fix the failure class, not the example. Trace each failed question to the data model, semantic layer or verification workflow. Then repair the underlying problem, such as a missing definition, an incorrect relationship or a missing guardrail.

Do not treat retesting as new evidence. Once a question has influenced the fix, passing that question shows only that the known case has been repaired.

Test the control on fresh questions. A general fix should catch the same problem across new metrics, segments and calculations. Once a fresh suite influences another repair, retire it and move to a new set of questions.

A fix is general only when it works beyond the question that revealed the problem.

The most recent held-out result was:

• Test size: 46 fresh questions, each run three times, giving 138 attempts on gpt-5-mini.
• Silent-error rate: 1.4%, or 2 of 138 attempts.
• Uncertainty: roughly 0.4% to 5.1% at 95% confidence, the range supported by a suite this size.
• Contested questions: both definitions were disclosed in all 42 attempts. When the definitions produced the same value, the system did not intervene.

For context, an earlier fresh suite had a 3.6% silent-error rate before two guardrails were generalised. The suites contained different questions, so this is not a controlled before-and-after comparison.

The held-out result is the one that matters. A rate measured on questions used to build the fixes shows how well the system learned those questions, not how reliably it handles new ones.

The same discipline applies to evaluation itself. Answer keys, graders and provider failures can all distort the result. The evaluation path is production code. If it fails, the result is not evidence. Store the question, expected outcome, model and configuration versions, trace, individual checks, latency and cost. When a summary number moves, the team should be able to explain which failures moved with it.

Start with the smallest reliable system#

A team starting now does not need every component from these experiments. It needs the smallest version of each responsibility, built around real questions from one business domain.

For each layer, one question tells you it is holding:

Start with one important domain. Do not begin with a giant catalogue, a giant prompt or a broad model comparison. Collect the questions people already ask, and establish a trustworthy path for those questions end to end. Add a mechanism only when a trace reveals a distinct failure it can own.

Who owns it, and what it takes. For one domain, an analytics engineer owns the trusted data and definitions. A governance owner resolves contested definitions. A backend or ML engineer owns the harness and evaluation.

With a deliberately small scope, a trustworthy first version can be built in weeks. Start with a thin, well-defined data model and resolve contested definitions during the build, not afterward.

Route each failure to the right backlog. A wrong grain is a modelling backlog item. A competing definition that fires repeatedly is a governance decision with a measurable cost. A clarification that users always resolve the same way is evidence that the answer belongs in the governed layer. Each routing follows the same repair method: locate the failed primitive, then move its grounding to the layer that owns it.

Security cuts across all four layers. The agent can use only approved tools and data the user is allowed to see. The same rule applies when comparing competing definitions. If the user cannot access the second result, the harness can say that another governed answer exists without exposing its value. Evaluation should test these boundaries. Knowing that an answer exists does not mean the user is allowed to see it.

The system around the model is the product#

Models will improve. They will retrieve more accurately, write better SQL and recover from more mistakes without help. Those improvements raise capability. They do not remove the need to decide what the business terms mean, which tools are allowed, what must be verified, or how silent failures slip through without verification.

So I no longer ask whether an organisation has an AI-ready warehouse or whether a particular model is accurate enough. I ask four questions:

1. Can the underlying data be trusted?
2. Are the business definitions executable and owned, and can the system tell when a request falls outside them?
3. What does the harness verify before serving?
4. Can the evaluation see the failures a user would miss?

Those questions produce an engineering plan, a governance agenda and a rollout decision. A model score alone produces none of them.

The model produces the answer. The architecture determines whether that answer deserves to be trusted.