A single AI agent conversation can look perfect and still be broken, leaders from LangChain, Conviva and CoreWeave said at VB Transform 2026
Our take
The recent discussions at VB Transform 2026 highlighted a critical, and perhaps surprisingly subtle, flaw in how enterprises are evaluating AI agents. The core issue isn't whether individual conversations appear successful – often they do, scoring well based on isolated interactions – but whether the aggregate performance of these agents delivers a positive user experience. This realization is driving a significant shift away from evaluating individual “traces” and towards a more holistic approach: comparing cohorts of users against a baseline. It's a move that echoes concerns raised in a recent piece about the limitations of safety guardrails, where we saw how even robust defenses can be bypassed [Safety guardrails blocked Hugging Face's defenders, not the attacker, when an AI agent breached its systems]. Understanding this nuance is paramount as organizations increasingly integrate AI agents into their workflows, and it’s a concept that extends beyond just evaluating the agent itself, as another article explored regarding the skills needed for aspiring computer scientists entering the AI era [Am I focusing on the wrong skills as a CS student in the AI era? (Need brutally honest advice) [D]].
The shift towards cohort analysis, as championed by speakers like Conviva's Hui Zhang, underscores a fundamental limitation of traditional evaluation methods. Scoring individual interactions, whether through LLMs or agent-as-judge systems, provides a myopic view. Zhang’s retail example—a seemingly successful shoe purchase flagged by unusually high clarification ratios and post-conversation abandonment—perfectly illustrates this point. These anomalies, invisible in isolation, become glaringly obvious when viewed across a larger user base. The call for a “second data source” – data encompassing the pre-, inter-, and post-conversation experience – is equally crucial. This broader perspective allows for a more nuanced understanding of the agent's impact and identifies category-specific issues that would otherwise remain hidden. This also validates the "evals are like the new PRD" concept discussed, where evaluation criteria evolve into a living specification, reflecting the ongoing needs and performance expectations of the agent.
Beyond the methodology of evaluation, the discussion also touched on the practicalities of implementation. CoreWeave’s Emmanuel Turlay’s approach of starting with the most capable model and working downwards, coupled with LangChain’s focus on fine-tuning models to detect user-perceived errors, demonstrates a pragmatic approach to resource optimization. The revelation that even simple techniques like regexes can serve as effective guardrails further highlights the importance of tailoring solutions to specific needs. The emphasis on cheaper, narrower "judge" models isn't a compromise on quality, but a strategic refinement of resources, allowing for broader monitoring and more targeted evaluations. This echoes the broader trend of adapting to the realities of AI deployment, where cost-effectiveness and scalability are just as critical as raw performance.
Ultimately, the conversation at VB Transform 2026 suggests that the future of AI agent evaluation lies not in achieving perfect scores on isolated interactions, but in establishing robust, always-on monitoring systems that can detect systemic issues across user cohorts. While human oversight remains essential, particularly in high-stakes domains, the increasing sophistication of AI-powered evaluation tools promises to significantly reduce the burden on human reviewers. The question remains: as AI agents become increasingly integrated into our lives, how will we ensure that these systems are not only intelligent but also reliably aligned with human values and expectations, and what new metrics will emerge to truly measure that alignment?
A single AI agent conversation can look flawless scored on its own and still point to a broken product. That gap is driving a shift in how enterprises evaluate agents, away from scoring individual traces and toward comparing cohorts of users against a baseline.
At VB Transform 2026, Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, described that shift, along with a parallel move toward cheaper, narrower judge models.
Agent-as-judge — judging one AI agent's output with another — hasn't replaced LLM-as-judge, which Chase said remains the default. The larger tension, Zhang said, is between automated judging, whether by LLM or agent, and human review.
"You have scalable but ungrounded, whether it's agents as judge or LLMs as judge, you grade the outcome, you grade the work. It still is very difficult to ground it and then you use humans and that's just not scalable," Zhang said. "The whole industry is facing this, which poison you want to pick."
Evaluation criteria now function as the product spec
That gap — a conversation that scores well but still signals a broken product — is what teams try to close by building an exhaustive evaluation suite before they ship anything. Chase said that doesn't work.
"We sometimes see teams that have almost eval paralysis," Chase said. "They're like, this is an eval set, I can't launch it. The best teams launch and then iterate."
Chase framed evaluation criteria as a living specification, not a one-time test suite: a product requirements document — the standard software-development spec for what an application should do. "Evals are like the new PRD," he said. "They define what your agent should and shouldn't do."
Turlay described hitting the same failure from a different angle. "I was trying to reach 100% coverage for my tests, and I still had bugs in production," he said — a test suite that looked complete but still missed what mattered, the same gap Chase was describing with evals.
Broad, always-on monitoring, he said, catches more real failures than an exhaustive pre-launch test suite. Teams should set up wide online checks first, use those to identify failure classes as they occur, then build a targeted offline evaluation set around the problems that surface.
Why scoring traces one at a time is a mistake
Even a well-built evaluation process can still score the wrong thing. Zhang's objection is to how most teams run evaluation: sampling traces, whether 50 of them or a full population, scoring each in isolation. That approach misses a signal that only shows up when comparing cohorts of users against a baseline, a method Zhang calls contrastive analysis.
Zhang illustrated it with a retail example: a shopper asks an agent for a running shoe ahead of a half marathon, the agent asks qualifying questions, and the shopper buys a shoe. Scored individually, that interaction looks fine. But the clarification ratio, how many follow-up questions an agent asks before completing a task, came in three times higher than baseline for that shoe category across the full user population. A second metric, how often shoppers finished their purchase outside the conversation, was five times higher than baseline for the same category.
Neither number is visible from a single trace. Both point to a debuggable, category-specific problem. Zhang said the industry also lacks a second data source: what happens before, between and after the conversation, not just the trace itself.
Sizing the judge to the job
Once contrastive analysis flags which category is actually broken, the next problem is what watches for it going forward — and at what cost. Turlay's rule was to start with the most capable model available to prove a task is solvable, then work down. If it can't be done with a top-tier model, he said, it won't work with a smaller one. Once a pattern proves viable, teams can sample a fraction of traffic instead of judging every interaction, and move simpler tasks like binary classification to smaller open source models.
LangChain took that further, fine-tuning its own model to detect when a user believes the agent made a mistake, a signal Chase calls perceived error. "The model we fine-tuned was a Qwen model," he said, referring to Alibaba's open source family. Combining hand labeling with distillation, the result performed well. "Same as [Claude]Sonnet, for, depending on how we served it, either 10 to 100x cost reduction," Chase said.
Not every guardrail needs a model. Chase pointed to Claude Code's own guardrails as proof: regexes, the common programming technique for finding and validating patterns in code. "A lot of the guardrails they had were just regexes," he said. "They weren't small LLMs, they were just regexes."
LLM-as-judge doesn't mean human-in-the-loop disappears
The bigger question is whether using LLM as a judge removes the need for a human in the loop.
Turlay pointed to accountability, drawing on his prior work at a self-driving car company. His team compressed data intake and retraining into a two-week cycle for shipping a new model to the car. Even then, someone still had to sign off.
"I felt confident on behalf of the company to say this model should go into the car," he said. The same logic extends to legal, finance and healthcare. "Before we can remove a human to say, I endorse this and I take responsibility legally for it, it's going to be a while before agents can do that on their own."
Zhang agreed a human has to remain the guardian on corner cases, even as automation eventually runs at a scale that beats individual human accuracy — machines can see more at the pattern level.
Chase went further: that human check isn't just a safety net. "Human in the loop is really important for building trust in how these agentic systems work, and also really important for memory and learning from systems," he said. "There has to be interactions in order for the system to learn."
Read on the original site
Open the publisher's page for the full experience