Enterprise AI is entering an evaluation gap: Agents are gaining autonomy faster than companies can verify them
Our take
The rapid expansion of AI agents within enterprises is encountering a critical bottleneck: a widening "evaluation gap." As highlighted in a recent VB Pulse survey, companies are granting AI agents increasing autonomy even as their confidence in the automated testing designed to govern them erodes. The findings, while directional rather than precise due to the self-selected sample, reveal a stark reality—half of enterprises have deployed agents that passed initial evaluations but subsequently caused customer-facing failures, with a significant portion experiencing these failures multiple times. This trend underscores a larger issue, as demonstrated by 57% of enterprises have watched AI agents be confidently wrong. The fix is an agentic context layer, but who has one?, where agents exhibit unwarranted certainty despite factual inaccuracies. It's a scenario mirroring observations around GPU utilization, where Wall Street is debating the AI buildout. Enterprises just answered: 86% say their GPUs run at half capacity or less, suggesting a deployment pace outpacing the development of necessary control mechanisms.
The core of the problem lies in the inherent complexity of agent evaluation. Traditional software testing focuses on predictable input-output relationships, but agents operate with a degree of freedom, choosing their own steps and tools. A successful agent run doesn't guarantee consistent success, as demonstrated by Anthropic's distinction between occasional brilliance and reliable performance. Enterprises are already recognizing this limitation, with a significant portion distrusting automated evaluations due to poor alignment with real-world outcomes – a far more pressing concern than issues of speed or cost. This aligns with the broader thesis of “ship agents first, controls later,” a pattern that will likely necessitate a significant retrofit cycle in the coming year as organizations prioritize systems enabling governable and dependable agent deployments. The NIST Generative AI Profile reinforces this, emphasizing the need for field testing, ongoing monitoring, and escalation processes to bridge the gap between controlled environments and real-world deployment conditions.
The solution isn’t to halt the pursuit of autonomy; the economic incentive for doing so is too powerful. Instead, a shift in focus is required. Enterprises must treat repeatability and regression testing with the same urgency as deployment speed. Every production incident should be incorporated into a permanent regression test suite, transforming support cases into learning opportunities. Low-risk tasks can justify greater autonomy, but critical functions—financial transactions, customer communications, and code deployments—demand stricter thresholds, consistent testing, and clear human oversight. The survey’s finding that larger companies are both adopting zero-human deployment at a faster rate and experiencing more failures highlights a crucial warning: embracing autonomy without robust assurance simply automates uncertainty, potentially amplifying the consequences of errors. The launch of tools like OpenAI introduces ChatGPT Work, a cloud-based AI agent that manages tasks across email, Slack and calendars illustrates the ambition of the space, but also underscores the imperative for responsible deployment and rigorous testing.
Looking ahead, the key differentiator won't be who deploys agents fastest, but who masters the art of predictable and reliable performance. The challenge isn't to eliminate humans from the loop entirely, but to strategically define which tasks can benefit from autonomy and to establish robust safeguards that prevent automated errors from cascading into significant business disruptions. As AI agents become increasingly integrated into core operational workflows, the question becomes not *if* failures will occur, but *how effectively* organizations can detect, mitigate, and learn from them to build truly dependable AI systems.
Enterprise AI teams are giving agents more freedom at the same moment their confidence in automated testing is collapsing.
Half of enterprises have deployed an AI agent or LLM feature that passed internal evaluations and yet still caused a customer-facing failure — one in four more than once — according to the June 2026 VB Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees.
The sample is self-selected rather than a probability sample, so the findings should be read as directional, not precise.
But enterprises are not responding by slowing automation: 66% of respondents already permit some production deployment without human review or are building systems intended to do so within the next 12 months. Only 5% say they fully trust the automated evaluations that would make those release decisions.
That mismatch is the evaluation gap: the autonomy ceiling is rising faster than the assurance beneath it.
It also fits a broader thesis that will be explored at VB Transform 2026: enterprises ship agents first, while the control layers around identity, evaluation, cost, context and orchestration are arriving later. The next year will be a retrofit cycle, with buyers shifting budget toward the systems that make agentic deployments governable and dependable.
Why a passing evaluation is not a working agent
Traditional software testing usually asks whether a defined input produces an expected output. Agent testing is harder because the system may choose its own sequence of steps, call tools, retrieve data, alter state and respond differently from one run to the next.
An agent can make several individually plausible decisions and still reach the wrong result. It may retrieve the correct account but update the wrong field. It may draft a valid refund request but send it without approval. It may call five tools successfully before a sixth step leaks sensitive information or leaves a workflow incomplete.
The survey shows enterprises already recognize this limitation. The most common reason for distrusting automated evaluation is poor alignment with real-world outcomes, cited by 29% of respondents. Bias or inconsistency follows at 21%, lack of explainability at 18%, and data leakage or privacy concerns at 17%.
That hierarchy matters. Enterprises are saying the score often does not predict what happens when a customer, employee or business process encounters the agent in production — not that automated scoring is too slow or expensive.
NIST makes a similar point in its Generative AI Profile: measurements gathered in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context and operating conditions. Its guidance calls for field testing, post-deployment monitoring and clear processes for escalating failures.
Capability is not consistency
A single successful run proves that an agent can complete a task. It does not prove that it will complete the task reliably.
Anthropic’s guidance on agent evaluation distinguishes between measuring whether a system succeeds at least once across repeated attempts and whether it succeeds every time. That distinction is essential for customer-facing or operational workflows. A model that occasionally produces an excellent answer may still be unacceptable if the same task fails unpredictably on the next attempt.
Enterprise teams should therefore treat repeatability as a first-class metric. That means running the same scenario multiple times, varying phrasing and context, testing tool failures, and measuring whether the final business outcome remains correct even when the route changes.
The evaluation set also has to evolve. Every production incident should become a permanent regression test. Customer escalations, failed tool calls, incorrect approvals and data-handling mistakes should feed back into the pre-deployment suite rather than remaining isolated support cases.
Autonomy should expand by risk, not by ambition
The survey does not imply that every agent action should require a person. Human review cannot scale across millions of low-consequence decisions.
But zero-human operation should be earned by demonstrated reliability and bounded by the consequences of failure.
Low-risk actions such as drafting internal summaries or categorizing documents can tolerate broader autonomy. Financial transactions, customer communications, code deployment, access-control changes and data deletion need stricter thresholds, repeated consistency tests, policy checks, rollback mechanisms and clear human escalation paths.
The risk isn't evenly distributed by company size, either. Larger enterprises — those with 2,500 or more employees — are moving toward zero-human deployment fastest, at 70% versus 64% for smaller companies, and they're also shipping more agents that go on to fail a customer, at 54% versus 48%.
That is the warning for enterprise leaders. Removing the human from the loop does not remove uncertainty. Without stronger assurance, it converts uncertainty into an automated production decision.
The market will keep pushing toward greater autonomy because the economic incentive is real. The organizations best positioned won't be those that remove people fastest — they'll be the ones that treat repeatability and regression testing as seriously as deployment speed.
Read on the original site
Open the publisher's page for the full experience