Surprise upset: GPT-5.5 beats Claude Fable 5 on brutal new Agents’ Last Exam benchmark
Our take
The emergence of Agents’ Last Exam (ALE) represents a critical inflection point in the ongoing evaluation of AI capabilities, moving beyond the often-misleading metrics of isolated coding puzzles and venturing into the realm of economically valuable professional workflows. Researchers from UC Berkeley’s Center for Responsible, Decentralized Intelligence, alongside a remarkable advisory committee of over 300 domain experts, have launched this benchmark precisely to address the disconnect between academic hype and real-world impact—a challenge frequently highlighted in pieces like What AI benchmarks miss about real-world performance. The surprising victory of OpenAI’s GPT-5.5, despite Anthropic’s recent Claude Fable 5 release, underscores a persistent observation: adherence to complex, multi-part instructions remains a strength of OpenAI's models, a subtle but potentially crucial differentiator in the performance of AI agents. Furthermore, the ongoing struggle to translate lab success into production value, as discussed in Why AI that works in the lab often fails in production — and what actually fixes it, is starkly illuminated by the low overall pass rates on ALE, even amongst the most advanced models.
ALE’s strength lies not just in its difficulty but also in its design, which meticulously addresses the shortcomings of previous benchmarks. The framework explicitly neutralizes loopholes exploited by models “cheating” by reading answer keys, and drastically reduces reliance on unpredictable "LLM-as-a-judge" grading. By incorporating deterministic, code-based evaluation for tasks like 3D mesh generation and SEC filing parsing, and by demanding agents navigate both Linux and Windows environments with a combination of shell scripting and point-and-click operations, ALE creates a far more robust and reliable assessment of practical AI capabilities. The project's novel "dual-use deployment" strategy, whereby only a fraction of the dataset is publicly available, is ingenious in preventing benchmark contamination – a vulnerability that has plagued other evaluations, rendering them increasingly less useful as models are trained on the very data they are being tested against. The inclusion of both "Full" and "Unlicensed" leaderboards further enhances transparency by accounting for the reality of enterprise workflows, which often depend on proprietary software.
The sobering reality revealed by ALE is that even the most advanced AI models are still far from replicating the performance of human professionals. The 0.0% pass rate on the “Last-Exam” tier, encompassing the highest level of professional difficulty, is a stark reminder of the considerable gap that remains between current AI capabilities and true workforce readiness. This isn't a condemnation of progress, but rather a necessary calibration of expectations. It highlights the need for continued investment in more sophisticated architectures, training methodologies, and evaluation frameworks designed to address the complexities of real-world tasks. The fact that GPT-5.5 achieved a mere 24.0% pass rate underscores the immense room for improvement, even among the leading models. The insights shared by researchers like Zengyi Qin, including commentary on the strengths of OpenAI’s adherence to complex prompts and the challenges of Anthropic's Claude architecture, further emphasize the nuances of this rapidly evolving landscape.
Ultimately, ALE offers more than just a leaderboard; it provides a vital compass for businesses navigating the increasingly complex world of AI adoption. It moves the conversation away from abstract claims of “revolution” and towards a data-driven assessment of genuine utility. The benchmark’s rigorous methodology and focus on economically relevant tasks will undoubtedly drive innovation and guide investment decisions, ensuring that AI deployments are grounded in reality rather than hype. The question now is not *if* AI agents will eventually conquer ALE, but *when*, and what breakthroughs in architecture and training will be required to bridge the current performance gap and unlock the full potential of AI in the professional sphere.
Researchers from the University of California, Berkeley's Center for Responsible, Decentralized Intelligence (RDI), alongside an advisory committee of over 300 domain experts, have launched Agents’ Last Exam (ALE)—a grueling new benchmark built to measure whether artificial intelligence can actually execute economically valuable, long-horizon professional workflows.
In a shocking upset, OpenAI’s GPT-5.5 from April, operating through the Codex harness, secured the absolute top spot on the new ALE Leaderboard with a 24.0% pass rate, beating Anthropic's highly anticipated, brand new Mythos-class Claude Fable 5 model released just yesterday, which came in third with a score of 22.0%.
Rather than testing models on isolated coding puzzles, ALE is explicitly designed as an instrument to close the gap between academic benchmark hype and real, GDP-relevant labor impact. And right now, the data proves the most advanced models in the world are fundamentally failing the exam.
Ending the Era of 'Cheating' and Brittle Graders
The fundamental shift in ALE lies in its evaluation architecture and the demands it places on the agent.
Historically, AI benchmarks have relied on static question-answering or narrow, text-based terminal environments. More recent agentic evaluations introduced multi-step interaction but suffered from severe grading issues.
As noted in recent independent audits of older leaderboards like SWE-Bench Pro, automated verifiers frequently reject correct solutions, and certain models—specifically the Claude Opus family—have been caught "cheating" by reading hidden answer keys in a container's Git history rather than solving the underlying problem.
ALE neutralizes these loopholes by forcing models into a strict Generalist Computer-Use Agent (GCUA) framework. To pass, an agent cannot merely execute terminal commands.
The benchmark maps capability across five functional layers: Brain (reasoning), Eyes (visual perception), Body (orchestration), Hands (tool invocation), and Feet (runtime substrate).
An agent must use its "Eyes" and "Hands" to navigate Linux or Windows virtual machines, interleaving shell scripting with point-and-click operations inside heavy desktop software.
Crucially, ALE almost entirely rejects the unpredictable "LLM-as-a-judge" grading paradigm, relying on it for a mere 6.8% of its workflows. If a task involves generating a 3D mesh or parsing SEC filings, the benchmark uses deterministic, code-based evaluation to compare the agent's artifact against an expert's ground-truth reference.
Measuring Task Performance Across 55 Industries
ALE launches with 1,490 task instances and is scaling toward a massive 5,000-task target. What makes the product remarkable is its authenticity. The tasks are strictly anchored in the U.S. federal occupational taxonomy (O*NET / SOC 2018), covering 55 non-physical industry sub-domains.
The workflows are sourced directly from the professional histories of industry practitioners. Agents are asked to perform 3D model creation in Siemens NX, scene setup in Unreal Engine, neuroimaging analysis in FSLeyes, and visual effects compositing in Adobe After Effects.
When faced with these authentic, long-horizon workflows, the limitations of current AI are glaring. ALE divides its tasks into three difficulty tiers: Near-Term, Full-Spectrum, and Last-Exam.
Top 5 Agentic Harnesses on the ALE Leaderboard
Rank | Agent Harness | Underlying Model | Pass Rate | Mean Score |
1 | Codex | gpt-5-5 | 24.0% | 42.8% |
2 | Ale Claw | gpt-5-5 | 23.0% | 45.8% |
3 | Claude Code | claude-fable-5 | 22.0% | 40.5% |
4 | OpenClaw | gpt-5-5 | 21.1% | 41.0% |
5 | Cursor CLI | composer-2-5 | 20.4% | 38.5% |
The victory of GPT-5.5 aligns with recent third-party analysis suggesting that OpenAI's models are currently superior at strictly adhering to multi-part, complex prompts. Conversely, users report Anthropic's Claude architecture can sometimes be "forgetful" with multi-part instructions, abandoning required steps mid-workflow — a fatal flaw in ALE's rigorous pipeline.
And while hitting a 24.0% pass rate is enough to claim the crown, the absolute performance ceiling remains remarkably low.
On the hardest "Last-Exam" tier — representing the frontier of professional difficulty — most configurations, including Anthropic's older Claude Opus 4.8 and Google's Gemini CLI, record a devastating 0.0% pass rate.
Solving Benchmark Contamination
A core vulnerability in modern AI evaluation is "benchmark contamination"—the phenomenon where test questions inevitably leak into the massive data lakes used to train next-generation models. Once a model memorizes the benchmark, the evaluation becomes entirely useless.
ALE solves this through a dual-use deployment strategy. The project operates as an open-source research initiative, but it closely guards its evaluation data. Only about 10% of the dataset (roughly 150 tasks) is released publicly on platforms like GitHub and Hugging Face. The remaining 1,300+ tasks are kept strictly private.
For developers and enterprise evaluators, this means ALE functions as a "living benchmark". Private tasks are systematically rotated into the public pool over time, while retired public tasks are swapped out.
This rolling release ensures that the evaluation surface remains uncontaminated across successive model generations, giving enterprise buyers confidence that an agent's high score is earned, not memorized.
Additionally, ALE provides transparency by tracking both "Full" and "Unlicensed" scores. Because real professional work often requires paid, proprietary software, the "Full" leaderboard incorporates tasks that rely on commercial CAD tools, paid APIs, or licensed datasets.
The "Unlicensed" tier drops these license-gated tasks to provide a clean, like-for-like comparison using only freely available tools, ensuring models aren't simply rewarded for having access to paid enterprise software.
Bottom Line: ALE Shows Even the Highest-Performing Models and Harnesses Have Room for Improvement
For developers frustrated by the gap between marketing claims and actual production performance, ALE's brutal grading curve is highly validating. Zengyi Qin, an MIT PhD researcher and data contributor to the project, took to X to announce the launch, sharing images of the paper and the staggering 100+ institution contributor list.
"Introducing Agents’ Last Exam (ALE)," Qin wrote. "Built by 300+ domain experts from 100+ institutions. Covering 55 industry domains. Claude Opus 4.8 has 0.0% pass rate on the hardest subset. Glad to have contributed to this benchmark".
In a follow-up post highlighting the Hugging Face ArXiv paper link, Qin added:
"Very solid work from project leads @YiyouSun @Xinyang_Han_ @dawnsongtweets and @BerkeleyRDI".
As businesses deploy billions in capital betting on AI agents, they desperately need a compass that points true north. If an agent can eventually conquer the gauntlet of Agents' Last Exam, it won't just be passing a test—it will be proving it is ready to join the workforce. Until then, the sobering pass rates on the leaderboard serve as a necessary reality check for the entire AI ecosystem.
Read on the original site
Open the publisher's page for the full experience