The Evidence Graph: Teaching an AI Analyst to Show Its Work

The product question was simple: when an AI analyst gives an answer, how can the user know what is safe to trust?

The obvious answer is explanation. Ask the analyst to explain itself.

But explanations are not evidence. A model can write a persuasive paragraph around a number it barely checked. It can describe a real query result, then attach a conclusion the result does not support. It can say "because" and still give you a real number for the wrong question.

So the work became less about better prose and more about a different object behind the prose.

I built an evidence graph: a typed map of the answer. It records each claim, the query result or earlier claim it depends on, and whether that support actually resolves.

The goal is not to read the model's thoughts. The goal is to let the user inspect the evidence behind each claim.

Short version#

The unit of trust is the claim, not the answer. A good analytics answer is a bundle of measurements, comparisons, caveats, conclusions, and causal hints. The graph lets the harness inspect each piece separately.

Claim support is a fourth metric. Coverage, silent error, and balanced accuracy need benchmark questions with known answers. Claim support can be checked on real production questions, because the harness only asks whether each declared citation resolves against the trace.

The graph turned vague model failures into semantic-model defects. Ambiguous metric names, unsupported dimensions, missing grains, and weak causal edges stopped being private model mistakes. They became broken or mislabelled claim support the data team can fix.

Repair improved inspectability, not magic correctness. In the repair loop, the model sends an answer with claims, the harness audits the claim citations, and any exact fault goes back to the model before the answer is served. That can fix malformed citations. It does not teach the analyst the right business answer.

The most important failure was flat reasoning. In 1,036 of the 1,405 answers that carried a graph, 73.7%, the analyst cited every measurement and never represented the diagnosis as a claim at all. That is the product lesson, and it is not a small one: a fully sourced answer can still be missing its argument, and the fourth metric scores it as grounded anyway.

Trust is not an explanation#

The previous experiment tested refusal: can the analyst say "I do not know" when there is no honest answer?

Guardrails helped. Silent error fell from nearly half of everything the analyst was asked to about two or three in a hundred.

But that still leaves a harder question:

If the analyst does answer, and the number is real, which part of the answer can be trusted?

Not the whole paragraph.

A paragraph is too coarse. It can put a checked number and an unchecked interpretation in the same sentence:

Active users rose 6.0%, but value moments fell because users opened reminders less often.

The first part may be easy to verify. The second part may not be.

That is why the evidence graph exists.

What it is: a small map behind the answer. Each node is one claim. Each edge says what that claim depends on: a query result, a semantic-model rule, or another claim.

Why it exists: it stops one real number from making the whole paragraph look proven.

What it is for: it lets the harness and the user see which parts are supported, weak, or missing support before trust attaches to the answer.

One caveat: a tick means the support link resolves. It does not mean the conclusion is true.

The graph can show what a diagnosis rests on. It cannot prove the inference is sound. That limit should be visible in the UI.

The old harness checked one thing: the final declared value. That was enough for lookup questions. It was not enough for diagnostic answers, because a diagnostic answer is not one claim.

An answer is a bundle of claims#

The evidence graph changes the unit of inspection from the answer to the claim.

A claim is a small assertion the analyst is willing to stand behind. It can be one of two broad kinds.

A measurement claim cites a field in a governed query result. For example, the claim might be c1: "days per user fell 16.4%." The evidence is r1:days_per_user.pct_change: r1 is the handle for the semantic-layer query result, days_per_user is the governed metric, and pct_change is the field that contains the value. The harness can check that deterministically. Did result r1 exist? Did it expose that field? Did the value match?

A reasoning claim cites earlier claims. For example: "value moments fell because users came back less often" is not a field in a query result. It needs premises. If the analyst wants to say that, it has to point to the measurement claims that establish days per user fell, active users rose, and the metric tree says days per user is part of value moments.

That boundary matters because it closes a loophole.

Without the boundary, the model can smuggle an argument inside a measurement:

Days per user fell 16.4%, which proves the pricing change caused the drop.

The number might be real. The causal jump is a different claim. If the graph only records that the number existed, the unsupported conclusion rides along for free.

This is the part I did not fully appreciate before building it. The graph is not just a prettier trace. It is a grammar for analytical accountability:

• measurements cite results
• conclusions cite measurements
• causal claims inherit the strength of the semantic model edge behind them
• anything that names no support is visibly unsupported

The user does not have to trust the paragraph as a whole. They can check the support claim by claim.

The evidence graph#

Every successful tool result gets a handle: r1, r2, r3. If the result contains several values, the model has to cite the specific field: r1:active_users.pct_change, not just r1.

A call that errors or gets blocked gets no handle at all, which is deliberate: there is nothing to stand on, and letting a claim cite a blocked call would make a refusal look like support. So a claim leaning on a query that failed has nowhere to point, and it audits as broken.

That is the behaviour I want, and it is also where the sharp edge is. A call that half succeeds, real rows, but for a narrower window than the question implied, does get a handle, and every citation into it resolves perfectly. The partial-period trap in the box below is exactly that shape, and the audit sails straight past it. This check catches missing evidence. It does not catch wrong evidence, and no amount of citation discipline will make it.

The answer then carries a list of claims. Each claim has either:

• sources: query-result fields or governed statements
• premises: earlier claim ids, such as c1 and c2

Carrying both is itself a defect. A claim is either read from evidence or derived from other claims. Mixing the two hides the distinction the graph is trying to make.

Concretely, an answer to "why did weekly value moments drop?" comes back carrying a list like this: the harness numbers the claims c1, c2, c3 in the order they arrive.

[
  { "text": "Days per user fell 16.4%",
    "sources": ["r2:days_per_user.pct_change"], "value": -16.44 },

  { "text": "Active users rose 6.0%",
    "sources": ["r2:active_users.pct_change"], "value": 5.98 },

  { "text": "Value moments fell because users came back less often",
    "premises": ["c1", "c2"] }
]

Two measurements that each name one field in one governed result, and one conclusion that names the claims it follows from and no data at all. The audit walks the list, resolves every sources entry against the run trace, checks each stated value is the one it cited, and checks every premise points at an earlier claim.

The important thing is where the judgement lives.

The model declares the claims. The harness checks whether the declared support resolves. If a claim cites r2:foo and r2 has no such field, the harness does not need a judge, a rubric, or a prompt. It can say the citation is broken because the citation is broken. That part of the audit is fully deterministic.

That is why this is different from ordinary "add citations" work. A document citation points at text, so the next question is entailment: does this paragraph support that sentence? That usually needs another model. A query-result citation points at a typed result. The first check is just lookup.

Where this comes from, because it is not mine. The shape — an agent building a typed graph of claims, results, and the links between them — is the subject of VeriGraph (Jin and colleagues, June 2026), which reports 87.61% on a claim-level grounding measure close enough to the one below that the resemblance is worth stating out loud. GroundEval (Flynt, June 2026) argues the same case for scoring an agent against its own trace instead of asking a judge model. Further back, the groundedness and faithfulness metrics in RAGAS, ALCE and most evaluation stacks have been breaking answers into claims and checking them one at a time for years.

So the idea is in the water. What I think is actually different here is narrower: those systems check claims against documents, so the question is entailment and it takes a model to answer. This one checks against a governed query result, so the question is a dictionary lookup. That is the whole source of the determinism, and it is the only part I would defend as new.

Protocol is the third axis#

The previous two experiments already had two axes.

Grounding varied what the analyst knew: raw tables, a star schema, a semantic layer, verified examples, a metric tree.

Guardrails varied what the analyst was allowed to do: refuse, check coverage, use governed tools, prove provenance, pass a verifier.

The evidence graph forced a third axis into the architecture, but the naming is easy to mix up.

Here is the clean split.

Protocol is the contract. It says what shape the model must return: an answer, a set of claims, citations for measurement claims, premises for conclusion claims, and sometimes a purpose for tool calls. Protocol is not what the user trusts directly. It is the API between the model and the harness.

The evidence graph is the artifact. It is the object created from that protocol: result nodes, claim nodes, edges from claims to evidence, edges from conclusions to earlier claims, and audit statuses on those edges.

Audit and repair are harness behavior over the graph. The audit checks whether each edge resolves. Repair uses one audit failure, an unresolved citation, to hand the answer back before it is served.

This sounds like bookkeeping. It is not.

Claims should not be treated as just another guardrail level.

A judge asks, "Is this answer right?" The claim protocol asks a different question: "Did the answer name what each assertion depends on?" Those two controls can be turned on or off independently.

So when I say protocol is the third axis, I do not mean "protocol" is a separate product feature from the evidence graph. I mean the evidence graph needs a model-visible contract, and that contract needs to be varied independently in experiments.

In this implementation, the evidence-graph system has four protocol switches:

• purpose: governed calls can carry a one-line reason
• claims: the answer declares one claim per assertion
• repair: broken claim citations are handed back to the agent loop
• rendered: measurement sentences can be written by the harness from cited values

Grounding and guardrails are mostly scored with lab metrics. They need gold labels to say whether the analyst was right. Protocol creates a different class of metric: whether the served answer can be inspected.

The fourth metric#

The first refusal article used three headline metrics.

Coverage asks: of the questions that have an answer, how many did the analyst attempt?

Silent error rate asks: across everything asked, how often did it hand over a confident wrong answer?

Balanced accuracy asks: did it do the right thing on both piles, answerable and unanswerable?

Those are the right metrics for a benchmark, but they all need gold labels. Someone has to know which questions have answers and what those answers are.

The evidence graph adds a fourth metric with a different job:

Grounded-answer rate asks: of the served answers that carried a claim graph, how many can be checked end to end from that graph?

This does not say the answer is true. It says the work is inspectable. That difference matters enough that I would never average it with the first three.

How to read those numbers. Each column is 195 attempts: 65 questions asked three times (31 that have an answer, 34 written to be refused). The analyst is gpt-5-mini at minimal reasoning, with gpt-5-mini at low reasoning as the verifier.

Grounded-answer rate is computed only for answers that carried a claim graph: 82 of 94 in the claims column, 90 of 95 with repair. No graph means no audit, not a failed audit.

An answer counts as grounded only when every claim can be followed to the value it cites. One broken claim fails the whole answer, because the reader can no longer inspect the argument end to end. The run traces are in the harness repo.

The first column is important because it has no evidence graph. It is not "bad" on grounded-answer rate. It is not measurable. There are no claim nodes to audit.

Adding claims creates the trust surface. Now the harness can ask: did the answer cite evidence that actually exists? With claims alone, 87.2% of graphed answers were checkable end to end. The harness also found 19 broken citations across 10 answers.

Repair made the agent more cautious and more grounded. The harness handed broken citations back to the agent loop before serving. Grounded-answer rate rose from 87.2% to 94.7%, silent error fell from 3.1% to 1.5%, and broken citations fell from 19 to 6. Coverage fell from 90.3% to 88.2%, which is the caution: the agent abandoned a few more questions rather than serve answers it could not support.

The cost was token consumption. Output tokens per attempt rose from 232 to 389. Counting input, cache discounts, and published gpt-5-mini prices, the full protocol cost about $0.0025 per attempt, up from $0.0019. Latency rose too, from 9.8 to 12.6 seconds at the median. The trade-off is clear: better grounded-answer rate and more inspectability for more tokens and a little more time.

The semantic model became debuggable#

The evidence graph also changed how failures looked.

Before, a bad answer was often just "the model got it wrong." After claim audit, the defect became more specific:

• it cited a value that did not exist
• it cited a real value but labelled it as the wrong metric
• it composed a number across metrics the governance layer does not license
• it used a point-in-time metric as if it had a period
• it asked for a dimension or segment the semantic model does not support
• it made a causal claim where the metric tree only had weak influence evidence

That matters because most fixes should not be prompt fixes. They should be model fixes: clearer metric names, safer member vocabularies, explicit coverage windows, declared grains, owned causal edges.

The clearest example was mislabelled metrics. In the repair column, 31 of 95 graphed answers had this problem, about a third of them.

The analyst was usually not inventing numbers. It was attaching real numbers to confusing governed names: value_moments, weekly_value_moments, and real_value_moments.

I kept this out of the grounded-answer gate. If mislabelled were part of that gate, grounded-answer rate would fall from 94.7% to 65.3%. I left it out because this is mostly a semantic-layer defect, not a citation defect. It still stays as a first-class audit finding, and that choice should be explicit.

That is the product lesson: the evidence graph is also a semantic-model debugger. It turns "these names are confusing" into a count and gives the team a concrete fix list.

There is one production catch. If a governed metric gets renamed, stored graphs that cite the old name can break. A real system needs versioning: an owner for each definition, tests that catch breaking renames, and a rule for stored graphs. Do they re-resolve against today's semantic layer, or stay pinned to the version they were built with? I have not built that yet. It is the next thing I would build before putting claim graphs in front of users.

The repair loop#

Once claims are typed, a new move becomes possible.

If the analyst returns an answer with unsupported citations, the harness does not have to serve it. It can hand the answer back and say exactly what is malformed.

Not "this seems wrong."

Not "please improve your reasoning."

Something much more mechanical:

Claim c3 cites r2:paid_search, but r2 has no paid_search field. Fix the citation or remove the claim.

This is a small self-healing mechanism. The model made a structured promise, the harness checked the promise, and the model got a bounded number of tries — two — to repair the broken part.

The result was specific. Compare the claims declared and claims + repair columns above: grounded-answer rate moved 7.5 points while balanced accuracy moved 0.6, which is the shape you would expect from a mechanism that checks citations and knows nothing about the business question.

That distinction is the whole point.

If repair had mostly improved correctness, I would be suspicious. A citation repair loop does not know the right business answer. It only knows whether a declared source exists. The fact that it mostly moved grounded-answer rate is exactly what the mechanism predicts.

That is why the graph is useful outside a benchmark. The harness can inspect a served answer from the trace alone: which claims are supported, which conclusions name premises, and which parts have no support to follow.

Cited facts are not a diagnosis#

The graph can make a fact trustworthy. It can show that a number came from a real query result or a governed metric.

But a "why" answer needs more than facts. It also needs a diagnosis.

That diagnosis is its own claim. It is the sentence that says what the facts mean: what moved, what probably drove it, and what the user should pay attention to.

The failure mode is easy to miss in prose. An answer can cite every number correctly and still leave the actual diagnosis unsupported. The reader sees real evidence beside the paragraph and may assume the whole answer has been checked.

The evidence graph makes that gap visible. It separates facts that were measured from conclusions that were argued.

The product rule is simple: do not call a diagnostic answer fully grounded just because the numbers are cited. If the facts are supported but the diagnosis is missing or unsupported, the UI should say that plainly:

Supported facts. Missing argument.

A "why did this move?" answer is inspectable only when the conclusion itself is represented as a claim and its support can be followed.

What I would ship#

In a real AI analyst UI, I would not show the user the entire trace by default. Raw traces are built for developers. They are too noisy for decision work.

I would show the answer and the evidence graph next to it. Something like this:

For every claim, the user should see:

• the claim text
• whether it is a measurement, conclusion, or causal statement
• the query result or claim premises it cites
• whether the support resolved
• whether the support is exact or correlational
• whether the semantic model has a gap the user should route back to the data team

Trust is observability#

The goal is not to make the user believe the model. The goal is to make the analyst's work observable enough that trust has somewhere to attach.

That is how we trust software systems. We trust the system around the service: logs, traces, checks, alerts, and people who can see where a failure happened.

An AI analyst needs the same surface:

• supported claims should open to evidence
• broken citations should be caught before the answer is served
• weak causal claims should be shown as weak
• repeated unsupported claims should become semantic-model backlog

This is not trusting the model alone. It is trusting a system: the semantic layer defines what numbers mean, guardrails control what paths are allowed, the protocol makes claims inspectable, and the evidence graph shows that inspection to the user.

The evidence graph did not make the analyst perfectly honest. It made honesty observable. That is the first version of trust I would ship.