At Waymo, an AI project isn't ready until its evals are — not when the model performs well
Our take
Waymo’s journey in deploying AI for autonomous vehicles offers a compelling case study for any enterprise venturing into the realm of AI agents. Few companies face stakes as high as Waymo, the self-driving car company under Alphabet that spun out of Google. Its models don’t simply generate text or automate back-office tasks; they guide vehicles through unpredictable streets, react to human drivers, and make critical, real-time decisions in the physical world. This reality highlights the importance of robust deployment strategies, and Waymo’s approach, detailed at VB Transform 2026, provides a valuable blueprint. As Mark Zuckerberg recently stated, Meta also sees a large enterprise opportunity extends beyond agents, demonstrating the growing recognition of AI's potential across various industries, though the inherent risks differ significantly. The emphasis on rigorous evaluation, something Target acknowledges as crucial—they say their real AI moat isn't the models — it's everything built around them—resonates deeply, particularly as AI moves beyond experimentation and into mission-critical applications.
The concept of "eval-forced development," where evaluation becomes a core engineering principle rather than a final validation step, is a particularly insightful takeaway. Waymo’s experience underscores that simply achieving high performance scores on training data is insufficient. Continuous evaluation, spanning training, post-training, simulations, and real-world driving, is crucial. Furthermore, these evaluations must be tied to tangible business outcomes, not just generic benchmarks. They've moved past simply testing for success, and are now proactively testing for failure conditions – a recognition that the true measure of an AI agent’s reliability lies in its ability to handle the rare, yet potentially catastrophic, scenarios. This echoes the challenges faced by companies like Bright Machines, who are tackling a major AI infrastructure bottleneck by ensuring efficient compute and storage utilization, a necessary component for sustained and comprehensive evaluation processes. The fact that model quality is only as good as the evaluation data underscores the need for careful data curation and the inclusion of diverse, representative datasets, specifically targeting those edge cases that highlight vulnerabilities.
Waymo’s deliberate inclusion of human oversight, even with advanced AI systems, is a vital reminder that accountability cannot be abdicated. The decision to release software and expand service areas remains firmly in the hands of human safety leaders, acknowledging that even the most sophisticated AI systems are not infallible, especially when human lives are at stake. This approach highlights a crucial tension: the drive for efficiency and automation versus the undeniable need for human judgment and control. The company's focus on data efficiency, prioritizing the most valuable training examples over sheer volume, demonstrates a pragmatic understanding of resource constraints—a reality facing most enterprises as AI adoption scales. The evolution from transformers to generative multimodal models further illustrates Waymo’s commitment to pushing the boundaries of AI capabilities while maintaining a rigorous focus on safety and reliability.
Ultimately, Waymo’s story is a powerful testament to the importance of building trust in AI systems. Earning that trust requires a holistic approach that encompasses not only powerful models but also a clearly defined objective, robust evaluation data, continuous testing, efficient infrastructure, and, crucially, accountable human decision-makers. The rise of agentic AI promises transformative gains in productivity and efficiency, but the lessons learned from Waymo—prioritizing continuous evaluation and human oversight—are essential for navigating the complexities and mitigating the risks inherent in deploying AI agents at scale. A key question moving forward is how organizations, particularly those lacking Waymo’s extensive resources and safety expertise, can effectively implement these principles to ensure the responsible and beneficial integration of AI into their operations.
Few companies face higher stakes when deploying AI than Waymo, the self-driving car company under Alphabet that spun out of Google. Its models do not merely generate text or automate back-office tasks: They help vehicles navigate unpredictable streets, respond to human drivers and make split-second decisions in the physical world.
But the methods Waymo uses to manage those risks — continuous evaluation, carefully curated data, human oversight and clearly defined business outcomes — offer a broader playbook for enterprises deploying AI agents in nearly any industry.
Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, explained at VB Transform 2026 how the autonomous vehicle company trains, tests and deploys AI at scale. To date, Waymo has driven more than 220 million fully autonomous, or "rider-only," miles, with 17 times fewer serious crash injuries than human drivers over the same distance, according to the company.
To achieve these impressive results, Joshi said Waymo has adopted what she called “eval-forced development” or “eval-centric development,” making evaluation a core part of engineering rather than a final check performed before deployment.
“The stage at which our projects are maturing can be easily kind of transpired based on the eval maturity that they showcase,” Joshi said.
In practice, Waymo assesses a project’s readiness partly by examining the maturity of the tests surrounding it. That approach has clear implications for enterprises building customer service agents, coding assistants, financial systems or other AI applications: If a company cannot reliably measure a system’s performance, it may not be ready to place that system into production.
Evals must continue after launch
Joshi said much of Waymo’s quality work has shifted toward evaluations, including tests conducted during model training, after training and inside open-loop and closed-loop simulations.
“Eval is not a one-time task to launch a model,” she said.
Waymo instead treats evaluation as a continuous process spanning driving, simulation and validation. Its methodology combines datasets, performance metrics and infrastructure capable of operating efficiently at scale.
For enterprises, that means testing an agent before launch is insufficient. Teams must continue evaluating it as underlying models, business processes, user behavior and incoming data change. Those evaluations should also connect to actual business outcomes rather than relying solely on broad industry benchmarks.
Joshi cautioned that model-quality measurements are only as trustworthy as the evaluation data behind them. Waymo therefore pairs its performance claims with information about the properties of the datasets used to test its systems.
Testing the rare and dangerous cases
Waymo’s evaluation hierarchy remains grounded in one overriding objective: safety.
The company draws on first-party driving logs, some third-party data and realistic simulations that expose its systems to scenarios spanning billions of synthetic miles. Task owners choose specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones and other complex environments.
The same principle applies outside autonomous driving. Enterprises need to test not only the routine requests their agents handle successfully, but also uncommon situations where errors could create financial, legal, security or reputational damage.
Joshi emphasized that Waymo does not leave release decisions entirely to automated systems. Its production-readiness reviews include extensive human oversight, while internal safety leaders approve software releases and service-area expansions.
“This is not AI-driven and completely automated and zero human oversight,” she said. “Human lives are at stake.”
Efficiency cannot come at the expense of reliability
Waymo faces another problem familiar to enterprise AI teams: Demand for compute, storage, memory and network capacity is growing faster than the resources available.
The company pursues efficiency across data extraction and storage, distributed model training, model distillation, simulation and evaluation. It also emphasizes “data efficiency,” selecting the most useful training examples instead of treating greater volume as inherently better.
Waymo began using transformers in 2017 and subsequently expanded into large language models, vision-language models and vision-language-action models. Joshi said the company now uses generative multimodal models as part of its foundation-model strategy.
Waymo divides its technology between onboard systems inside each vehicle and off-board infrastructure used for model development, data processing and simulation. That combination forces the company to optimize both real-time inference and the larger systems supporting it.
Agents need their own evals
Waymo also uses AI agents internally as productivity tools for engineers. Joshi said agents help analyze data distributions, assess data efficiency and triage problems found in vehicle telemetry, training runs and failed evaluation jobs.
The goal is to accelerate investigative work so engineers can devote more time to judgment and difficult technical problems. But Waymo also evaluates those agents to ensure they produce trustworthy, accurate results rather than sending employees down unproductive paths.
For enterprise leaders, Waymo’s larger lesson is that agentic AI requires more than choosing a powerful model. Organizations need a clearly defined objective, representative evaluation data, continuous testing, infrastructure that can operate efficiently and named human decision-makers who remain accountable for deployment.
"Earning trust is supremely important," Joshi said.
Read on the original site
Open the publisher's page for the full experience