What I Look For in a Data Platform Audit

The fastest audit I run takes about ten minutes and one question.

I ask two people from two teams for the same number. How many active users did we have last month. Finance says one thing, Product says another, and they are not close. Nobody in the room looks surprised, and that lack of surprise is the actual finding. If a company can't agree on how many users it has, everything built on top of that number is decoration.

That is what a data platform audit really measures. Not whether the stack is modern. Whether the company can trust its data enough to decide with it. A shiny warehouse full of contradictory numbers is worth less than a boring one everybody believes.

The first number tells you everything#

The trust question comes before any of the technical ones, because it decides whether the rest even matters. If people don't trust the numbers, they won't use them to decide, and a platform nobody decides with is just an expensive archive.

So I look for the tells of low trust, and they're consistent:

• two teams report different numbers for the same metric, and both have a story for why theirs is right;
• executives quietly re-validate a dashboard by hand before they'll act on it;
• analysts burn their week reconciling definitions instead of answering questions;
• business users export to a spreadsheet and rebuild their own version of the truth;
• nobody can tell you which dashboard is the source of truth without hedging.

None of that is a data-quality problem in the technical sense. The pipelines can be green. It's an operating problem: the numbers exist, and the company has collectively decided not to believe them. Everything downstream of that decision is theater.

It's almost always the definitions#

When I pull the thread on a diverging number, it almost never ends at a broken pipeline. It ends at a definition nobody ever agreed on.

This is the finding that repeats across every audit I've done. Finance counts active users one way, Product another, Growth a third, and each has been quietly correct inside its own logic for years. No one built it wrong. No one built it together. Airbnb hit the same wall and wrote about it plainly: when the CEO asked something as simple as which city had the most bookings last week, "Data Science and Finance would sometimes provide diverging answers using slightly different tables, metric definitions, and business logic." That was Airbnb, with some of the best data engineers on earth. It is happening at your company too.

The metrics that cause the most damage are the ones that look too obvious to define. Revenue. Active user. Conversion. Retention. Churn. Every one hides a pile of decisions: gross or net, which unit of analysis (user, account, company, transaction), are internal and test accounts stripped out, what window counts as "active," is the definition written down anywhere or does it live in one analyst's head and three slightly different SQL files.

Modern is not the same as useful#

Most stack reviews ask the wrong question. The one I usually get hired to answer is phrased as "is our platform modern enough," and that's the wrong end of it.

Benn Stancil, who co-founded Mode, put the fashion problem better than I can: "my favorite definition of the modern data stack is that it's data tools that launched on Product Hunt." Snowflake, dbt, Fivetran, a semantic layer, a reverse-ETL tool, an observability vendor. You can assemble every fashionable component and still have three teams computing revenue three ways, because none of those tools decides what revenue means. That's a human agreement, and no purchase order buys it.

Bryan Offutt at Locally Optimistic said the quiet part directly: "you can't out-tool poor organization." That line is most of my job. The migration everyone wants to talk about is rarely the thing holding the company back. Ownership, definitions, and a review cadence are, and none of them ship in a vendor contract.

The rest of what I check, fast#

Trust and definitions are where the real leverage is, but a full audit walks the rest of the platform. I move through it quickly, looking for a handful of things.

Does the model hold the logic, or does the BI tool? Many BI problems are modelling problems wearing a costume. If every dashboard carries its own hidden formula, the company has as many definitions of a metric as it has charts, and it will lose control of its numbers the moment the analyst who wrote them leaves. I want models organized around business concepts, not source tables, with the transformations legible and the assumptions visible. The dbt crowd has been right about this for a decade: analytics is software, so "bad data can lead to bad analyses, and bad analyses can lead to bad decisions," and the fix is the boring software discipline of version control, tests, and ownership.

Is it reliable when leadership opens it? Not every dataset needs to be real-time. Every dataset needs expectations. Barr Moses coined "data downtime" for the periods when data is "missing, erroneous, or otherwise inaccurate," and her five pillars, freshness, volume, distribution, schema, and lineage, are a decent checklist for whether anyone would notice a break before the CEO does. I'll add one thing the tooling misses: most breaks start upstream, at the seam Chad Sanderson calls treating a "database schema as a non-consensual API," where an app engineer changes a column with no idea the finance dashboard depends on it. Observability tells you it broke. It doesn't tell you whose deploy did it.

Does anyone actually own it? Every important metric and pipeline needs a business-aware owner, not just an on-call engineer. Without one, "who is responsible for this number" stays a shrug, and trust decays a little every time a definition changes and nobody announces it.

Is the team building leverage or fighting fires? A technically strong team can be trapped in reactive reporting, closing tickets and never building anything that compounds. Schario and Murphy's line is the test: without intentional space, a data team is "unlikely to be anything more than ticket closers." I look at intake, prioritization, and how much of the week is recurring support versus durable work.

Can leadership actually see the business? The last check is the simplest. Sit a founder in front of the reporting and ask them to tell me what's growing, what's declining, which segments matter, and where revenue quality is slipping. If they can't answer from the system, the platform is not doing its job, however modern it is.

But can you even measure that?#

An honest objection: if I'm judging a platform by the decisions it drives, how do I measure that? Barry McCardel at Hex argues you can't, and that trying is a trap. "So, how do you measure data team ROI? You don't," he writes, because the attribution is impossible and the exercise curdles into measurement theater.

He's right that the spreadsheet is a trap, but his conclusion goes a step too far. I'm not chasing a dollar figure attributed to the platform. I'm looking for evidence that a number changed a decision: a pricing call that moved because someone finally trusted the margin data, a channel that got cut because the cohort view held up. McCardel half-agrees, from the other side: the ROI "will effectively round to zero unless they inspire action." So I audit for whether a number ever changed what someone did.

The output is a plan, not a complaint list#

A bad audit ends in a two-hundred-page catalogue of technical grievances that everyone nods at and no one owns. That report is where audits go to die.

A good one produces a prioritized 30/60/90 plan with a name on every line: the current-state diagnosis, the trust and definition gaps, the modelling and reliability risks, the ownership holes, the quick wins, and the deeper architecture work, each assigned to a human who is accountable for it.

And the very first item is almost always the same. Lock one number. Pick the metric the company argues about most, get the three teams in a room, and agree a single definition, owned and written down. It's unglamorous, it fixes nothing technical, and it does more for trust than a quarter of platform work. Once people believe one number, they tend to believe the next one faster, and the rest of the plan gets a lot easier to land.

Final thought#

A data platform isn't valuable because it holds a lot of data. It's valuable when the company can look at it, believe it, and act. That takes more than pipelines and dashboards. It takes a decision spine: trusted metrics, models that hold their logic, clear owners, and reporting wired to the way the business actually runs. The audit is just the honest look at whether that spine is there, or whether the company has been mistaking a modern stack for one.