A Self-Improving Analytics Stack That Does Not Grade Itself

An AI analyst that is right on launch day does not stay right. I have seen it happen in a deployment I worked on. The agent passed its evals (test questions with known correct answers), and people started to use it. Over the following weeks the answers got worse. The agent had not changed. The tables and metric definitions under it had, and nobody updated the notes it reads about them.

Anthropic reported the same pattern in its analytics agent: accuracy fell from about 95% at launch to about 65% within a month. It is back at about 95% now that the team treats keeping those notes current as engineering work.

The Thoughtworks article Making Your Data Ready for Agentic AI states the rule for the repair: "Fix the definition, not the prompt." Ian Macomber's The Shape and Feel of the Post-AI Data Stack, written from his work on Ramp's internal research agent, puts a feedback loop at the centre of the stack. Every correction a data scientist makes, he writes, "only compounds if you build the infrastructure to catch the error, trace the process, and distribute the fix to your entire company". Neither article describes that infrastructure in detail, and this article is my design for it.

In the obvious version, one system finds the problem, changes the code and tests the change. In this design, the component that makes a change never writes or sees the test for it.

The design in one picture#

The design has two parts, which the diagram calls planes. The answer plane handles one question in seconds. The improvement plane works over days. Black numbers mark the answer plane, copper numbers the improvement plane. When a number appears twice, both arrows belong to the same step. Step 0' happens once per session, before any question is asked.

Two arrows have no number because they happen at the same time as diagnosis (step 9). The Gap Coordinator drafts eval cases into the evals repo, and domain owners verify the expected answers.

Why the system must not grade itself#

A system that is optimised against a score can raise the score without fixing the problem. In the Darwin Gödel Machine experiments, an agent that could rewrite its own code got a perfect score on a hallucination check. It did not fix the hallucination. It changed what the check could detect. The authors call this objective hacking.

In analytics, an eval case is a question with a verified number. If one engine writes the fix, the question and the number, a passing test only shows that the engine agrees with itself.

Databricks keeps these roles apart in Genie, its AI analyst: benchmark questions are separate from the context the agent learns from. This design applies the same separation to every part of the loop.

Each component gets only the access it needs#

The MCP gateway gives each component only the access in this table.

The diagnostic engine reads widely because a wrong answer can start in any layer. A wrong revenue number can come from an outdated context file. It can also come from a metric that does not state its grain (what one row represents), a pipeline that failed last night, or an event the app never sends. The diagnostic engine looks at all of them and can replay the failing question in a test environment.

Macomber lists the questions his team asks when an eval fails: "Was the domain context ambiguous? Was the data model incomplete, or defined in code outside of the agent's purview? Did the user have the access they needed?" Each question points to a different layer. The fix for the last one is a permission, and for this reason the diagnostic engine also reads access logs.

The improvement engine cannot push to the main branch or approve its own pull request. Its code does not run in CI until a person approves the run. GitHub documents similar controls for its Copilot cloud agent. In the incidents I have read about where an AI agent damaged data, the agent had direct write access to production. One example is the Replit coding agent in July 2025.

Who writes the test cases#

There are three sets of test cases. All of them live in the evals repo, which neither engine can read. Take one gap as the example: the agent refused "What was net revenue in Germany last quarter?" because no metric excluded refunds.

The held-out set never includes a question used to build a fix, and the Gap Coordinator does not see it. Its score is the only one I would show leadership.

The registry starts with a few gap types: a missing or unclear definition, an outdated context file, missing data or a missing event, and a missing permission. Hamel Husain and Shreya Shankar recommend building failure categories from real traces, so I would revise these after reading the first hundred reports.

What counts as better#

Coverage is the share of answerable questions the agent attempts instead of refusing. A silent error is a confident number that is false: a wrong answer, or an answer to a question that has none.

Coverage alone is easy to raise: the agent just refuses less. So a change passes the release gate only if:

• coverage goes up,
• silent errors do not go up,
• no regression case fails.

After the change ships, three signals tell the registry whether the gap is closed:

• Semantic-layer share: the share of answers that went through the semantic layer instead of hand-written SQL. Anthropic calls it "the single most actionable number we tracked". A drop usually means a context file has drifted, or a new kind of question has appeared.
• Correction rate: how often people dispute an answer.
• Consensus divergence rate: Macomber's check at Ramp. Ask for each board-deck metric through every interface and model, and count the different answers. For board metrics it should be zero.

What I could not find, and what is unproven#

Several vendors now suggest fixes automatically. Snowflake, Databricks, and the BI tools Omni and Hex each change the semantic-layer definitions inside their own product. Anthropic and Grab go further: an agent drafts a pull request and a domain owner reviews it. Most of those pull requests change context files. I found no published system that traces a failed question back to a missing tracking event or a missing pipeline and fixes it there.

That part is also the least proven. Nobody has published how an analytics agent's failures split between missing or outdated context and data that was never collected. Anthropic's results point to context. If that holds, the registry and the release gate deliver most of the value, and the part that changes pipelines and tracking events can wait. I would measure the split before building it.

So the first parts I would build are the Gap Registry and the release gate. Neither of them needs write access to anything.