When Both Numbers Are Right

How many active users did we have last week?

Product's governed metric says 886. It excludes internal and test accounts because Product is trying to understand customer activity.

Platform's governed metric says 919. It includes those accounts because Platform is planning capacity and support volume.

Both metrics are governed definitions, computed by executable SQL and tied to legitimate decision processes. Both are right. Ask an AI analyst the question and it serves one of them, while nothing in the answer shows that it chose.

This is a different failure from hallucination. The agent did not invent a number or use an unapproved calculation. It returned a valid result from the governed layer. The failure is that it presented one legitimate interpretation as though no other interpretation existed.

This was not a duplicate-metric problem#

My previous experiment studied definition debt: old metrics beside new ones, several names for the same calculation, and hidden scope differences nobody intended to preserve. That kind of ambiguity can be found before an agent runs. Scan the definitions, identify the overlap, repair the model.

The active-users pair is different.

A definition with no owner and no consumer is probably a leftover. Neither of these is. Deleting either would break a legitimate decision process.

The difference is also small enough to look harmless. The metrics disagree on eleven of twelve tested slices, by between 0% and 5.17%, and by 3.72% on the week as a whole. Small differences are often the most dangerous ones: a value a few tenths of a percent from its sibling passes a range check and looks normal in a board slide.

The problem is not that the warehouse contains two numbers. The relationship between them is missing. The semantic layer knows how to compute each metric, but it does not say that they are competing definitions of one business concept.

The preflight scan can find the collision and identify the filter that separates the definitions. It cannot decide whether one definition should be removed or both should remain. That is a business decision. The owner and use of each metric tell us that this pair is legitimate and needs to be handled, not cleaned away.

I expected the agent to ask#

The first approach was instruction-driven clarification. The agent had both definitions, a clarification tool, and an instruction to use it when more than one definition fit. It still chose one.

The name moved the answer#

I wanted to know what drove that choice. I kept the question, SQL, filters, values, owners and descriptions fixed. Then I changed only the metric names.

Question: How many active users did we have last week?

A reliable response: ask whether internal and test accounts should be included, or show both values and explain the difference: 886 excluding those accounts and 919 including them.

The highlighted metric is the one the agent selected.

In Example 1, Product had the familiar active_users name, so the agent served 886. In Example 2, the familiar active_* wording moved to Platform, and the agent served 919. Nothing else changed. The answer moved with the name.

Catalogue order was not the cause: reversing it did not change the answer. Removing the exact name match made one model ask which definition the user meant, but the other models still chose one. Naming influenced the choice; it did not resolve the ambiguity.

More information did not make it ask#

Ideally, the agent should stop and ask whether internal and test accounts should be included. If it answers without asking, it must at least show both numbers and explain the difference.

I tried to make that happen in several ways. I gave the agent a clarification tool, told it to ask when several definitions fit, placed the definitions next to each other, changed their order, and showed it the exact SQL behind the result. The SQL even showed the filter that excluded internal accounts. None of those changes made the behaviour consistent.

I then ran a controlled test using 17 contested questions, with three attempts per question. In all 51 attempts, the agent had both definitions, the clarification tool, and an instruction to ask. It never asked and never showed both values. It served one metric every time.

The agent had

Product metric886

Platform metric919

Definitions, SQL and an instruction to ask

51 out of 51answered without asking→↓

The user received

One number

• No clarifying question
• No competing metric
• No visible choice

The descriptions were available. The agent could explain the difference when directly asked to compare the metrics. It had the information, but it could ignore it. Nothing checked whether it had considered both definitions before returning an answer.

The fix moved the choice out of the model#

The clarify-by-instruction approach depended on the model recognizing the ambiguity and choosing to expose it. I moved that responsibility into the harness.

The harness first finds governed definitions that compete for the same business concept. When the agent selects one, it executes both competing governed definitions at runtime with the same period, dimensions and filters, then compares the results.

Whether the definitions conflict depends on the question. They may produce different company totals but the same answer for a particular segment or calculation. That is why the harness compares their results at runtime instead of permanently labelling the metric ambiguous.

Four requests run through both governed definitions. The harness intervenes only where the two metric results differ.

Request	Product metric	Platform metric	Compared	Harness
Active users, last week	886	919	differ	AmbiguityShow both or ask
Active users on web	277	289	differ	AmbiguityShow both or ask
Active users on an empty platform slice	0	0	equal	No ambiguityContinue
A metric with no competing definition	one result	none	nothing to compare	No ambiguityContinue

Comparing the results told the harness when ambiguity existed. It still needed a rule for what to do next.

The five designs changed who was responsible for exposing the choice. The model could be instructed, blocked, given advisory context, checked, or bypassed when it omitted the disclosure.

I compared five designs on the same 17 ambiguous questions. The questions covered three competing metric pairs across several slices, plus a difference and a ratio built from those metrics. I ran each question three times with gpt-5-mini, giving 51 attempts per design.

A result counted as correct only if the agent asked the user to choose or showed both valid metric results.

Clarify tool#

This was the clarify-by-instruction approach: give the model a clarification tool, show it both definitions, and tell it to ask when more than one definition fits. The agent exposed the choice 0 times in 51. Every attempt served the metric whose name matched the question. The same design on gpt-5.6-sol, the stronger model, also scored 0 of 51.

Verdict: Clarify-by-instruction remained optional. The model could ignore both the instruction and the tool.

›Industry context: Cortex AnalystA documented example of clarify-by-instruction. Read it →

Snowflake documents the same general control pattern in Cortex Analyst. Its question_categorization custom instructions can tell the model that a question about users is unclear unless the user says which product they mean, and ask for that detail.

This is instruction-driven clarification: a human writes the ambiguity rule, and the model is expected to follow it. The Clarify tool result above tested the same dependence on model behaviour. In this experiment, providing the instruction and the clarification action was not enough. The later mechanisms added a different control: the harness executed competing definitions, compared their values, and verified or constructed the disclosure before serving the answer.

Read Snowflake's documentation.

Hard gate#

My first mechanism blocked the answer whenever the two values differed. It returned both definitions, both values and the filter that separated them, and the agent turned that into a good question in the user's words: should internal and test accounts be included, or not?

Signal

Values differ

886 ≠ 919

→↓

Harness

Block

Stop the calculation

→↓

Model

Respond again

Both results supplied

→↓

Cost

Interrupt

Even when scope is clear

The gate forced clarification on all 51 ambiguous attempts. But it also interrupted 10 of 12 questions where the user had already specified the metric.

Verdict: The gate caught every ambiguity, but also interrupted questions that were already clear.

Alternative attached#

The next design kept the runtime comparison but removed the block. The harness computed the competing definition with the same period and filters, then attached the result to the model's context. The disclosure was still advisory: the model could use it or ignore it.

[also] total_value_moments is an equally governed answer to the same question and returns a different number here: value_moments 2,435 against total_value_moments 2,675, 9.86% apart.

Model

Draft

886

→↓

Harness

Attach rival

Platform: 919

→↓

Model choice

Optional

Use it or ignore it

→↓

User

Receives

One or both results

The agent wrote that competing value into its answer 50 times in 51. The one miss was February's active users: it served 593, with the 614 sitting in front of it, and never mentioned it.

Verdict: Runtime comparison provided the right context, but disclosure still depended on the model using it.

Disclosure check#

The next design added verified disclosure. The harness examined the proposed answer before it was served. If it named only one of two metric results that differed, the answer went back once with both values.

Model

Draft

886

→↓

Harness

Check

Are both results shown?

→↓

If missing

Return once

Supply 919 again

→↓

Model retry

Still optional

One omission escaped

Two of the 51 first drafts omitted the competing value. The harness returned both to the model, and both came back correct.

Verdict: Verified disclosure caught omissions, but correction still depended on another model response.

Constructed disclosure#

The final design removed the retry. The harness deterministically constructed the disclosure by adding any omitted value itself. It already had the governed value because it had computed both definitions for the comparison.

Model

Draft

886

→↓

Harness

Detect omission

919 is missing

→↓

Harness

Complete

Insert the governed value

→↓

User

Receives

886 and 919

The disclosure check and constructed disclosure both handled 51 of 51 attempts. Each encountered two omissions and repaired both. The difference was responsibility: constructed disclosure did not depend on the model correcting itself.

Verdict: Disclosure no longer depended on model compliance. The harness inserted any missing value itself.

How the harness finds competing definitions#

Before the harness can compare two metrics, it needs to know which definitions might compete. It learns that in two stages.

First, preflight scans the semantic layer offline and records possible pairs. Then, when a question arrives, the harness runs the relevant pair with the same filters and compares the results. The offline scan finds where ambiguity might exist. The runtime check confirms whether it matters for this question.

Runtime ambiguity detector

narrow / expensive

On the live question, run both metrics and compare. Disclose only when the numbers actually differ.

Contested evals

targeted / slower

Held-out questions that land on contested definitions. Check it discloses both metric results without over-clarifying the rest.

Static candidate scan

broad / cheap

preflight scans the layer for definitions that could compete, same concept and grain but a different filter, and writes a candidate index in CI.

The offline scan builds the candidate index. preflight reads the semantic layer and finds definitions that describe the same business concept but differ by a filter or scope. For active users, that filter decides whether internal and test accounts are included. The scan records these pairs in a small index that is rebuilt in CI whenever the layer changes.

This tells the harness where a conflict might exist. It cannot tell whether the definitions will produce different values for a particular question.

The runtime check decides whether the conflict matters. When the agent selects a metric, the harness checks the index for a competing definition. If it finds one, it runs both with the same period, dimensions and filters, then compares the results.

If the results agree, the harness continues normally. If they differ, it intervenes. This avoids interrupting questions where the candidate metrics happen to produce the same value.

The two passes divide the work by cost. The static index narrows the field to a couple of candidates per metric, so the runtime check adds only a query or two per answer rather than re-deriving every definition. The runtime comparison then decides the case on real values, which is what keeps both the cost and the over-clarifications down.

The resulting workflow#

1. Find governed definitions that compete for the same business concept.
2. Use their owners and business uses to decide which conflicts to repair and which to keep.
3. Execute the competing definitions for the exact request.
4. Compare the values they produce.
5. If the values differ and the question has not resolved the choice, put the competing value in front of the model, and check that the answer asks or discloses both.
6. Add any missing value before the answer reaches the user.

Put together, the final runtime flow looked like this:

The architecture separates three jobs: find competing definitions offline, compare their values at runtime, then verify the answer and construct any disclosure the model omitted.

We did not teach the model to stop choosing by name. Its first answer became provisional. The harness checked it and completed any missing disclosure before it reached the user.

What it costs#

The expensive half runs offline: the static scan that builds the candidate index runs in CI, so it adds nothing to a live question. At query time the mechanism only acts on metrics that appear in the index. Every other question is untouched.

Detecting the conflict costs no model calls: it is a query and a comparison. The five designs above ran from 2.0 to 2.3 model calls per answer, and the shipped one, which also verifies the answer, sat at 2.3 against 2.2 for enforcing nothing. Because the check fires on sensitivity rather than membership, a question that touches a contested metric but whose two values agree pays only the extra query and is never interrupted. The latency a user feels is one extra warehouse query on contested questions.

The evaluation was missing an outcome#

Before this experiment, my questions had two states: an answer existed, or it did not. That was enough to measure answering and refusal. It could not represent the active-users question without declaring one governed definition correct and the other wrong.

The evaluation now sorts questions by how many valid answers each one has:

The third pile is the one this article is about, and it fails in two opposite ways. Serve one metric silently and the user never learns there was a choice: the invisible, expensive error. Ask the user to choose on a question whose two metrics produce the same value and you have interrupted them for nothing: the visible, cheap error, an over-clarification. Handling ambiguity well means catching the first without spending too much on the second, which is the trade-off the five designs were weighing.

The silent choice is the hard one to catch. The selected metric exists. Its SQL succeeds. Its number matches the result. Provenance and value checks pass. A reviewing model can also accept it, because nothing in the answer announces the missing alternative.

In the 51-attempt comparison, both verified designs handled every contested question. A later held-out suite confirmed the behaviour on fresh cases: no contested question slipped through as a silent choice, and questions whose two results agreed were left alone. The system still makes other mistakes. The full scoring model, the rollout metrics, and how I test the evaluation itself are in the reliability architecture piece.

What this means for a real analytics stack#

This failure crosses the same four layers as the broader reliability architecture:

For a team starting now, I would do five things:

1. Ask two departments for the same important number. If they disagree and both can explain why, treat the pair as governed disagreement rather than data debt.
2. Run preflight offline to find overlapping definitions. Delete those with no owner or consumer, then record the legitimate competing pairs that remain.
3. Compare those definitions at runtime using the user's exact filters and period.
4. Choose a product policy: stop and ask for high-stakes decisions, or disclose and verify both metric results for exploratory analysis.
5. Add several-answer questions to the evaluation before claiming the analyst is reliable.

Most companies already have disagreements like this. They sit in dashboard filters, team conventions, and context passed between analysts. An AI analyst makes them riskier because it can turn one team's convention into a company-wide answer without saying so. The goal is not to force every team onto one number. Legitimate disagreement can remain. Hidden choices cannot.

›Limits of this experimentWhat these results do and do not establish. Read it →

The experiments use a synthetic warehouse and a deliberately constructed set of contested definitions. The rates do not predict what another company will observe. They establish a narrower mechanism: when two governed definitions produce different answers to the same request, advice does not reliably expose the choice; an explicit comparison and verification flow can.

Each test case declared the expected outcome before the run. For a contested case, the grader accepted either a clarifying question or an answer that showed every governed value. Serving one value silently failed. Refusing also failed because valid answers existed. The check was deterministic, not another model judging the prose.