AI Analytics Reliability Lab.
Controlled experiments on what makes AI analysts reliable over data. Change one thing at a time, hold the model and questions fixed, and measure what it buys.
Start here
What It Takes to Trust an AI Analyst
Lessons from six experiments on building reliable AI analytics: trusted data, governed definitions, a bounded verification harness, and evaluation that can see silent failure.
Experiment 01 · Grounding
How much does structure buy?
Better grounding materially improves accuracy, but capability improves faster than honesty.
Read: Agentic Analytics: How Much Does Grounding Actually Buy You?
Experiment 02 · Reliability
Can the analyst safely say "I don't know"?
Silent error fell from 48.5% to 2.4% across the guardrail study.
Read: Agentic Analytics: Teaching an AI Analyst to Say I Don't Know
Experiment 03 · Protocol
Can an answer show what its claims actually rest on?
Citation repair improved grounded-answer coverage, while the evidence graph exposed missing reasoning structure.
Read: The Evidence Graph: Teaching an AI Analyst to Show Its Work
Experiment 04 · Repair
Which primitive failed, and where should it be fixed?
1,488 graded runs produced a repair matrix mapping failure types to the cheapest correct layer.
Read: The AI-Readiness Repair Matrix
Experiment 05 · Ambiguity
Can we predict wrong choices before the agent runs?
In the governed-metric subset, wrong-metric selection dropped from 24% to 0 after ambiguity repair.
Read: Why AI Analysts Pick the Wrong Metric
Experiment 06 · Contested
What does an AI analyst do when two governed definitions are both right?
Clarify-by-instruction: 0 of 51. Competing metric attached: 50 of 51. With a disclosure check: 51 of 51.
Read: When Both Numbers Are Right
Experiment 06 · Part 2 · Answerability
How does an AI analyst know which questions it can answer?
A closed-world ontology sorts each request into governed, computable, or uninstrumented, so existence and joinability become a graph check rather than a model guess.
Read: How an AI Analyst Knows What It Can Answer
Experiment 07 · Parameter grounding
Which parameters of a query can an AI analyst ground in the semantic layer, and what must the layer hold for the rest?
With the words people use modelled as data and every parameter of the query checked separately, 1 answer in 72 was a silent error, and 96% of answerable questions were answered. One run, development version; the final run is pending.
Read: What an AI Analyst Needs From Your Semantic Layer
New experiments, as they ship.
I publish each experiment on LinkedIn first, with the method and the numbers. Follow to get the next one.
Field notes, essays and everything else, newest first: all writing →