How an AI Analyst Knows What It Can Answer

Ask an AI analyst for 90-day customer retention and imagine that no retention metric exists.

That fact alone does not tell the system what to do.

The warehouse may contain customer signup dates and later activity, enough to calculate retention once somebody defines it. Or it may contain no history that can support the question. A search of the metric catalogue returns the same result in both cases: no match.

One question is computable but undefined. The other is unsupported by the data. An analyst should not treat them as the same problem.

This is the answerability problem: before deciding how to answer, the system must establish whether the required facts exist and whether they can be combined legitimately.

A catalogue can find definitions. It cannot prove absence#

A semantic layer is good at answering a positive question: which governed metric represents revenue, activation or active users?

It is weaker at answering a negative one: does the organisation have no way to calculate what the user requested?

The absence of a metric name is not enough. A useful calculation may be possible from governed measures that already exist. Conversely, the warehouse may contain columns with promising names that cannot be combined at a valid grain or through an approved join.

A language model is especially inclined to bridge this gap. It can find nearby concepts, write plausible SQL and substitute a familiar measure. That flexibility is useful during exploration. It is dangerous when the result is presented as a governed answer.

The system needs a factual way to distinguish “not defined yet” from “not in the data.”

An explicit list of everything the company does not collect will not work. The possible absences are unlimited, and the list will be wrong as soon as the warehouse changes. The practical answer is to define the present completely and derive absence from it.

Close the world at the serving boundary#

I built a graph of the analytical marts: their entities, attributes, measures, grains, governed metrics and approved relationships. The graph is generated from the semantic layer's own definitions rather than hand-authored, and every node is checked against the live warehouse schema, so it cannot drift from the metrics it must agree with.

Most of this graph already exists in the semantic layer. Cube's data graph and dbt's MetricFlow semantic graph already model entities, measures and curated join paths, and both refuse a join that was never declared; Unity Catalog foreign keys and Snowflake's semantic views carry the same relationship metadata for their own agents. The closed-world graph does not replace that model. It reuses it, and adds the one decision the layer does not make: whether a question with no governed metric is still computable from the measures and joins that already exist, or genuinely uninstrumented. That verdict, and the routing it enables, is the part that is new.

For the purpose of serving an answer, the graph is treated as complete. If a required input is not in the graph, the system treats that input as unavailable. If two entities have no approved path between them, the system does not invent a join.

This is the closed-world assumption. It does not claim that the organisation knows everything. It means that the agent may claim only what the verified analytical surface supports today. New data and relationships can be added later through the normal modelling and governance process.

The distinction matters. A table that has not been modelled can still appear as an isolated entity. Its columns exist, but the graph will not connect it to the rest of the business until somebody curates the relationship. An incomplete graph therefore tends to withhold an answer instead of inventing a join.

Three outcomes, not a yes or no#

The graph does not return a vague confidence score. It gives the harness one of three useful outcomes.

Governed#

An approved metric already answers the question. The system can execute that definition and apply the usual checks for filters, period, coverage and result provenance.

Computable#

The required inputs and join path exist, but no governed metric defines the requested calculation. For retention, that might mean the warehouse has signup dates, activity dates and a relationship through the customer, but no approved retention rule.

The graph has established possibility, not permission. A strict analytics policy can refuse with no_governed_definition. A controlled authoring workflow can propose a definition, verify the SQL and grain, and show the new definition with the answer. What it should not do is silently improvise a calculation and present it as established company meaning.

Uninstrumented#

A required input or relationship is absent. If the question needs support-resolution time and the analytical warehouse has no support events or timestamps, more prompting cannot produce an honest answer. The system should refuse and identify the missing data.

The graph establishes what is possible. Governance determines what is allowed to be served.

The model and the graph have different jobs#

Natural-language questions do not arrive as graph nodes. The model still has to interpret the request and identify its ingredients.

For “90-day customer retention,” it might identify a customer, a signup event, subsequent activity, a 90-day interval and a relationship connecting the events. That decomposition requires semantic judgement. A graph cannot infer it from the sentence alone.

The graph then checks whether those referenced ingredients exist and can be joined. That part does not require language judgement. The model should not be able to invent a node, approve its own join or turn a failed lookup into permission to use raw SQL.

This creates a clean seam:

• The model decides what the user appears to mean.
• The graph establishes whether the warehouse supports the required ingredients.
• Governance policy decides whether the resulting calculation may be served.
• The harness enforces that decision before the answer reaches the user.

The boundary is more reliable than asking the model to read a long description of the warehouse and judge answerability afresh for every question. The description is context. The graph verdict is a control.

What changed in the experiment#

No metric did not mean no data. The test question asked which acquisition channel had the best 90-day retention. The warehouse contained the signup cohorts, activity and channel data needed for the calculation, but the organisation had never defined a retention metric.

The old check correctly reported that no metric existed. But raw SQL was still available, so the agent kept going. It decided that retention meant “active exactly on day 90,” calculated a result and declared organic the best channel, even though it barely differed from referral. The answer did not show that the definition had been invented or that the apparent winner was effectively a tie.

The graph changed the classification. It confirmed that the required inputs existed and could be joined, so the request was computable rather than uninstrumented. That established what was possible. It did not give the model permission to invent a definition.

The verdict the harness routes on is a typed object, not a sentence. For this case:

{ verdict: "computable",
  reason:  "no_governed_definition",
  found:   ["dim_users.signup_date", "fct_user_days.active_date"],
  join:    "dim_users.user_id = fct_user_days.user_id",
  missing: ["governed retention metric"] }

Governance policy controlled what happened next. Under a strict policy, the system refused with no_governed_definition. Under a transparent policy, it could calculate retention only if the answer stated the definition used: “share of non-internal users active on day 90.” Otherwise the answer was returned for disclosure or clarification. In both cases, the agent could no longer silently invent the meaning behind the number.

Answerability is larger than the ontology#

The graph closes one important question: do the required analytical ingredients exist, and can they be connected?

It does not prove that the complete answer is safe to serve. Other controls still have work to do:

Calling all of these “answerability” can hide the architecture. The closed-world graph owns existence and joinability. Other components own coverage, computation, ambiguity and premise checking.

A computable verdict is answerability, not authorization. The graph decides whether the ingredients exist and can be joined. It does not decide whether this user may see the rows behind them. Row and column security and object grants still bind on the identity that runs the query, so a question can be computable for the warehouse and still refused for the person asking.

Build the smallest useful boundary#

Start with one analytical domain you already trust, not the entire warehouse.

1. Map what exists. Generate the entities, measures, attributes and grains from the semantic layer and schema.
2. Approve the joins. Two existing columns do not automatically make a valid relationship.
3. Check before exploring. Return governed, computable or uninstrumented before the agent reaches for raw SQL.
4. Keep the graph current. Rebuild it when the warehouse changes and refuse if the verified version is stale.
5. Name every gap. Send missing definitions, data and relationships to the right owners.

This turns refusal into useful work. A missing definition goes to the governance backlog. Missing data becomes an instrumentation task. A missing relationship becomes a modelling task. “I cannot answer” becomes an explanation of what the system needs before it can.

The model interprets the request. The ontology establishes whether the warehouse can support it. Governance decides whether the result may be served.

A trustworthy AI analyst does not need to know everything. It needs to know where its knowledge ends, and it must not cross that boundary silently.