An eval harness found what qualitative review couldn't: AI models are most confident when wrong
Our take
The recent article highlighting the discrepancy between LLM output sounding plausible and actually being correct resonates deeply with the challenges we're seeing across the enterprise AI landscape. Many organizations, eager to capitalize on the potential of large language models, are rushing to deploy tools that rely on these models without adequately validating their accuracy. As [I compiled Doom’s renderer into a 21B-parameter transformer -- no training anywhere [P]]( /post/i-compiled-doom-s-renderer-into-a-21b-parameter-transformer-cmstm62te0ex1mi9zut7yg9h8), we’ve seen firsthand the complexity of replicating even seemingly straightforward tasks, let alone ensuring absolute correctness. The author’s point about qualitative reviews – relying on human intuition to judge output – being insufficient is particularly crucial. It’s a common trap, especially when dealing with complex data and nuanced reasoning, and it leaves organizations vulnerable to deploying systems that generate confident, but ultimately wrong, answers. This is amplified by the rapid advancement in model capabilities; fluency and coherence are increasing, making it even easier to mistake plausible-sounding output for verifiable truth.
The shift from evaluating for fluency to demanding verifiable accuracy is a critical evolution. As LLMs move beyond simple productivity enhancements and begin to influence high-stakes decisions – guiding analysts, informing compliance reviews, or prioritizing operational responses – the consequences of error become significantly more severe. The distinction is stark: a suggestion that *seems* reasonable in a draft email is far less problematic than an AI-driven assessment that leads to a flawed legal judgment or a mismanaged system failure. This is why Arun Mishra’s emphasis on building evaluation harnesses, specifically those utilizing synthetic ground truth datasets, is so important. It’s a rigorous, albeit time-consuming, process, but it provides a quantifiable measure of accuracy that qualitative reviews simply cannot. The effort required to define "correct" within a specific use case – as Mishra notes – is a valuable exercise in itself, forcing a deeper understanding of the underlying problem and the model's role in solving it. Understanding how watermarks will work, as explored in Anthropic shares more details about how Claude’s new watermarks will work, also highlights the ongoing search for reliable verification methods.
The development of robust evaluation harnesses isn’t just about catching errors; it’s about building trust in AI systems. When users can see that a tool is consistently accurate, they are more likely to adopt it and rely on its insights. This, in turn, unlocks the true potential of AI to transform workflows and drive better outcomes. The challenges Mishra highlights – creating realistic synthetic datasets that mimic real-world complexity, designing scoring functions that accurately reflect the value of ranked output – are significant, but they are also solvable. The fact that the article focuses on the "hard part" – the synthetic dataset – underscores the importance of investing in this foundational element. It's a reminder that simply throwing more data at a model isn't a substitute for carefully crafted, targeted testing. The ease with which one can How to Build a Simple AI Web Scraper with Python provides a useful, if simplified, illustration of how this evaluation process can be approached.
Looking ahead, the rise of automated evaluation harnesses will be a defining trend in enterprise AI. We anticipate seeing a proliferation of tools and platforms that streamline the creation and management of synthetic datasets and scoring functions. The ability to continuously monitor and validate AI models in production – not just during initial development – will become a non-negotiable requirement for responsible AI deployment. The question isn’t *if* organizations will adopt these practices, but *when* and how effectively they will integrate them into their AI development lifecycle. The true measure of success won't be the fluency of the AI’s responses, but its demonstrable accuracy in delivering the right answers, consistently and reliably.
There is a step in the development process for large language model (LLM)-assisted tooling that most teams skip because it's tedious, time-consuming, and doesn't produce results visible to end users: Verifying that what the model is saying is actually correct. Not fluent, not coherent, not topically relevant — correct in the sense of accurately identifying the right answer to the specific problem the tool was built to solve.
The gap between "this output sounds right to me" and "this output is verifiably correct" is where most LLM-assisted enterprise tools fail quietly. They pass internal review because the output sounds right. They fail in production because those people weren't reviewing against ground truth — they were reviewing against their intuition about what a good answer looks like.
This distinction matters more as LLM-assisted tools move from productivity accessories to components that influence real business decisions. If your AI-assisted tool is shaping how an analyst investigates a data quality issue, how a compliance reviewer decides whether to escalate a flagged record, or how an operations team triages a validation failure — the accuracy of its output has real consequences. "Seems reasonable" is not an adequate evaluation standard for that.
What qualitative evaluation actually catches
The standard evaluation approach for LLM output in enterprise tooling is qualitative: A sample of outputs is reviewed by someone with domain knowledge, judged against a mental model of what a good answer looks like, and the prompt is adjusted if too many outputs seem off.
This catches a specific class of problems: Outputs that are obviously wrong, poorly formatted, or off-topic. These are real issues worth catching. They're also the easy ones.
What qualitative evaluation consistently misses is the class of outputs that are wrong in ways that are difficult to see without checking against something external. An explanation that confidently identifies the wrong root cause, in language that sounds authoritative, based on reasoning that sounds plausible — this passes qualitative review. It fails the moment someone with the right context checks it against what actually happened.
In a system whose value proposition depends on accuracy, "sounds plausible" is not the same as "correct." The two can diverge significantly, and qualitative review won't tell you when they have.
What an actual eval harness looks like
The alternative is building an evaluation harness that scores model output against labeled ground truth — a set of cases where the correct answer is known, against which you can measure accuracy rather than coherence.
I built this while developing a root-cause explainer for data migration drift: A tool that takes a detected drift event and generates a ranked explanation of what most likely caused it. The first prototype produced fluent, specific-sounding explanations that passed qualitative review. When I tested it against cases where I already knew the root cause, the explanation was wrong often enough to matter.
The eval harness I built works in three parts.
First, a synthetic ground truth dataset: Cases where the correct answer is known by construction. This meant introducing specific, controlled causes into a test pipeline — schema changes, transformation logic bugs, source system behavioral shifts — recording exactly what I introduced, and running the model against the resulting drift events. The correct answer for each case was the cause I had deliberately introduced.
Getting the synthetic scenarios realistic enough to be useful required more care than I expected. Early versions were too clean — the drift signal was obvious in ways that real production drift events aren't. Adding realistic noise, overlapping signals, and cases where multiple plausible causes were present simultaneously was what made the synthetic set actually predictive of real-world performance.
Second, a scoring function that evaluates ranked output. Binary correct/incorrect isn't sufficient when the model produces a ranked list of likely causes rather than a single answer. An explanation that correctly identifies the root cause as the third most likely candidate is meaningfully different from one that identifies it as the most likely. The scoring function evaluated two dimensions: Presence — did the correct answer appear in the output at all — and rank — how prominently was it featured relative to incorrect candidates. These were combined into a weighted score that rewarded both finding the right answer and ranking it appropriately.
Third, systematic evaluation across the full synthetic dataset rather than spot-checking. Running the harness across the complete set reveals patterns that spot-checking misses: Which categories of problem the model handles reliably, which it consistently gets wrong, and which combinations of signals produce the highest rate of confident incorrect explanations.
What the evaluation revealed
The results were more informative than any qualitative review could have been.
Schema change scenarios scored well — the model was reliable at identifying upstream schema changes when the evidence was present and distinctive. Transformation logic bugs were harder — the model consistently identified the right general category but misattributed the specific change that caused the problem, particularly when multiple changes had been made close together. Overlapping-signal scenarios were the hardest — cases where two different causes occurred close in time produced the highest rate of confidently wrong explanations.
That last finding is the one that qualitative review would never have surfaced. The model's expressed confidence didn't correlate with its accuracy — it was most confident in the cases where it was most wrong. Without the eval harness measuring against ground truth, that pattern would have been invisible.
The practical implication for enterprise AI deployment
For teams deploying LLM-assisted tools in enterprise contexts — particularly tools that influence how people investigate problems, triage alerts, or make routing decisions — the eval harness question to answer before production deployment is: Have we measured accuracy against cases where we know the right answer, or have we only reviewed whether the outputs seem reasonable?
If the answer is the latter, the tool has been tested for fluency and coherence but not for correctness. Those are different properties. For tools that shape business decisions, correctness is the one that matters.
Building the synthetic ground truth dataset is the hard part and the part most worth investing in. It forces you to define precisely what "correct" means for your specific use case — which turns out to be a useful exercise independent of the evaluation itself. The scoring function and the harness infrastructure are relatively straightforward once you have that definition. Without it, you're measuring something other than what you're trying to guarantee.
Arun Mishra is an enterprise architect.
Read on the original site
Open the publisher's page for the full experience