Real-World Web Tasks Trip Up Even the Best AI Agents

Introducing ClawBench, a groundbreaking benchmark that assesses AI browser agents on 153 real-world tasks across 144 live websites.

April 14, 20263 min readMachine Learning

The numbers are sobering, and they should be. ClawBench's findings that the best AI agent succeeds on only one in three real-world web tasks is not a failure of ambition; it is a measure of reality. When a leading model like Claude Sonnet 4.6 tops out at 33.3% on 153 everyday jobs across 144 live sites, we have to stop measuring progress in demos and start measuring it in durability. The gap between a controlled environment and a production website is not a minor variable. It is the entire problem.

What makes this benchmark worth your attention is not the leaderboard ranking, though that is instructive. It is the insistence on live websites and the five layers of behavioral data. Synthetic benchmarks reward models for pattern matching in a sandbox. ClawBench forces agents to contend with the messy, shifting, occasionally broken web that actual users navigate. The fact that a text-only model like GLM-5 lands second at 24.2% tells us something practical: visual grounding is helpful, but not sufficient. Reasoning and tool use matter more than we assumed. For teams evaluating AI agents, this is a reminder to ask not "Can it do a task?" but "Can it recover when the page changes, the button moves, or the login fails?"

The category splits reveal where the real work lives. Finance and academic tasks are relatively tractable, with the best model hitting 50%. Travel and development tasks are far harder. That asymmetry is a gift. It tells builders where to focus engineering effort and where to set user expectations. If you are deploying agents for expense reporting or literature reviews, you have a fighting chance today. If you are booking multi-leg trips or debugging a codebase, you are still in research territory. The request interceptor that blocks final HTTP calls before irreversible actions is a clever safety valve, but it also underscores how far we are from autonomous trust. An agent that needs a kill switch for payments is not ready to manage your finances.

Our take is simple: ClawBench is the kind of benchmark the field needs, not because it flatters any model, but because it exposes the distance between capability and reliability. The interactive trace viewer and step-level diagnostics are not just nice extras. They turn failure into a learning tool. For anyone building on these systems, the practical next step is to run your own tasks through ClawBench's open dataset and see where your agent stumbles. The path forward is not a better sales pitch. It is better failure analysis.

From Machine Learning

We introduce ClawBench, a benchmark that evaluates AI browser agents on 153 real-world everyday tasks across 144 live websites. Unlike synthetic benchmarks, ClawBench tests agents on actual production platforms.
Happy to answer any questions! We're actively looking for feedback on task selection and evaluation methodology.

Read the original at Machine Learning

Briefing

Source: Machine Learning
Published: April 14, 2026

Measure your coding agent against a new open-source benchmark suite.
Sharing an open-source benchmark suite (paper-lantern-challenges) that measures coding-agent performance with vs without retrieval-augmented technique selection across 9 everyday software tasks. Disclosure: I'm the author of the retrieval system under test (paperlantern.ai/code); the artifact being shared here is the benchmark suite itself, not the product. Every prompt, agent code path, and prediction file is in the repo and reproducible. Setup. Same coding agent (Claude Opus 4.6 as the planner, Gemini Flash 3 as the task model), same input data, same evaluation scripts across all 9 tasks: test generation (mutation score), text-to-SQL (execution accuracy), PDF extraction, contract extraction, PR review, text classification, few-shot prompt selection, LLM routing, summarization evaluation. Independent variable: whether the agent could call a retrieval tool over CS literature before writing its solution. One pass per task, no retries, no manual filtering of outputs. Task selection. Tasks were chosen to span the everyday-engineering surface a coding agent actually faces, not specialized ML scenarios. Selection criteria: (1) unambiguous quantitative metric, (2) baseline performance well below ceiling, (3) standard datasets where they exist, (4) eval reproducible on a free Gemini API key in roughly 10 minutes per task. Eval methodology. Each task uses its task-standard quantitative metric (mutation score for test_generation, execution accuracy for text_to_sql, F1 on labeled spans for the extraction tasks, weighted F1 for classification, etc.). Full per-task scripts and dataset choices are in the repo - one directory per task, evaluate.py as the entry point, README.md per task documenting methodology and dataset. Retrieval setup. The "with retrieval" agent has access to three tool calls: explore_approaches(problem) returns ranked candidate techniques from the literature, deep_dive(technique) returns implementation steps and known failure modes for a chosen technique, compare_approaches(candidates) is for side-by-side when multiple options look viable. The agent decides when and how often to call them. Latency is roughly 20s per call; results cache across sessions. The baseline agent has none of these tools, otherwise identical scaffolding. Comparability. Both agents share the same task-specific user prompt; the only system-prompt difference is the retrieval agent's tool-call grammar. Predictions and per-task prompts are diffable in the repo (baseline/ and with_pl/ subdirectories per task). Results. Task Baseline With retrieval Delta extraction_contracts 0.444 0.764 +0.320 extraction_schemas 0.318 0.572 +0.254 test_generation 0.625 0.870 +0.245 classification 0.505 0.666 +0.161 few_shot 0.193 0.324 +0.131 code_review 0.351 0.395 +0.044 text_to_sql 0.650 0.690 +0.040 routing 0.744 0.761 +0.017 summeval 0.623 0.633 +0.010 The test-generation delta came from the agent discovering mutation-aware prompting - the techniques are MuTAP and MUTGEN - which enumerate every AST-level mutation of the target and require one test per mutation. Baseline wrote generic tests from pretrain priors. The contract extraction delta came from BEAVER (section-level relevance scoring) and PAVE (post-extraction validation), both 2026 techniques that post-date the agent's training. 10 of the 15 most-cited sources across the experiments were published in 2025 or later, which is the conservative argument for why retrieval matters: the agent could not have reached these techniques from parametric memory. Failure modes. Self-refinement hurt text-to-SQL (the agent second-guessed correct queries after reading work on SQL ambiguity). Two suggested techniques (DyT, SeeDNorm) were architecture-incompatible in the autoresearch experiment and got discarded. Retrieval surfaces better options, not guaranteed wins. Reproducibility. Every prompt, every line of agent code, every prediction file, every eval script is in the repo. Each task directory has a README documenting methodology and an approach.md showing exactly what the retrieval surfaced and which technique the agent chose. Repo: https://github.com/paperlantern-ai/paper-lantern-challenges Writeup with detailed per-task discussion: https://www.paperlantern.ai/blog/coding-agent-benchmarks Happy to share additional design choices in comments. submitted by /u/kalpitdixit [link] [comments]