Measure your coding agent against a new open-source benchmark suite.

Introducing an open-source benchmark suite designed to evaluate coding-agent performance in real-world software tasks.

April 25, 20263 min readMachine Learning

Measure your coding agent against a new open-source benchmark suite. — Open-source 9-task benchmark for coding-agent retrieval augmentation. Per-task deltas +0.010 to +0.320, all evals reproducible [P]

A benchmark that measures whether a coding agent performs better when it can search academic literature before writing code is exactly the kind of concrete evidence this field needs. The results are clear: across nine everyday software tasks, the agent with access to retrieval outperformed the baseline in every single case, with deltas ranging from a modest 0.010 to a striking 0.320. This is not a theoretical argument about the future of AI-assisted development. This is a reproducible finding, published openly, with every prompt, evaluation script, and prediction file available for inspection.

What makes this benchmark valuable is its honesty about the practical reality of coding agents. The author deliberately chose tasks that mirror actual engineering workflows, test generation, text-to-SQL, PDF extraction, contract analysis, code review, classification, prompt selection, LLM routing, and summarization evaluation. These are not exotic ML challenges. They are the kinds of problems that developers actually bring to agents today. The selection criteria were refreshingly pragmatic: each task has an unambiguous quantitative metric, a baseline performance well below ceiling, standard datasets where available, and an evaluation that runs on a free API key in roughly ten minutes. This is a benchmark built for everyday use, not for conference leaderboards.

Some of the largest gains came from techniques published after the agent's training cutoff. The contract extraction improvement came from BEAVER and PAVE, both 2026 papers. The test generation improvement came from mutation-aware prompting strategies that enumerate AST-level mutations. The agent could not have reached these approaches from parametric memory alone. Ten of the fifteen most-cited sources across all experiments were published in 2025 or later. This is the conservative argument for retrieval: it is not about making agents smarter, but about making them aware of knowledge they fundamentally do not possess. Retrieval as a memory expansion, not an intelligence boost.

The author also documents the failures, which is rare and welcome. Self-refinement hurt text-to-SQL performance because the agent second-guessed correct queries after reading academic work on SQL ambiguity. Two retrieved techniques in the auto-research experiment were architecture-incompatible and had to be discarded. Retrieval surfaces better options, but it does not guarantee wins. The agent still needs to evaluate, filter, and decide. The benchmark makes this visible. If you are building or evaluating coding agents, this repo offers a replicable methodology for measuring whether your system benefits from the same kind of tool access. The data is in the repository. The methodology is documented per task. The opinion here is plain: this is how benchmarks should be built.

From Machine Learning

Sharing an open-source benchmark suite (paper-lantern-challenges) that measures coding-agent performance with vs without retrieval-augmented technique selection across 9 everyday software tasks. Disclosure: I'm the author of the retrieval system under test (paperlantern.ai/code); the artifact being shared here is the benchmark suite itself, not the product. Every prompt, agent code path, and prediction file is in the repo and reproducible.

Read the original at Machine Learning

Briefing

Source: Machine Learning
Published: April 25, 2026

When LLMs consult research papers, automated experiments gain a 3.2% edge.
Ran a controlled experiment measuring whether LLM coding agents benefit from access to research literature during automated experimentation. Setup: Two identical runs using Karpathy's autoresearch framework. Claude Code agent optimizing a ~7M param GPT-2 on TinyStories. M4 Pro, 100 experiments each, same seed config. Only variable — one agent had access to an MCP server that does full-text search over 2M+ CS papers and returns synthesized methods with citations. Results: Without papers With papers Experiments run 100 100 Papers considered 0 520 Papers cited 0 100 Techniques tried standard 25 paper-sourced Best improvement 3.67% 4.05% 2hr val_bpb 0.4624 0.4475 Gap was 3.2% and still widening at the 2-hour mark. Techniques the paper-augmented agent found: AdaGC — adaptive gradient clipping (Feb 2025) sqrt batch scaling rule (June 2022) REX learning rate schedule WSD cooldown scheduling What didn't work: DyT (Dynamic Tanh) — incompatible with architecture SeeDNorm — same issue Several paper techniques were tried and reverted after failing to improve metrics Key observation: Both agents attempted halving the batch size. Without literature access, the agent didn't adjust the learning rate — the run diverged. With access, it retrieved the sqrt scaling rule, applied it correctly on first attempt, then successfully halved again to 16K. Interpretation: The agent without papers was limited to techniques already encoded in its weights — essentially the "standard ML playbook." The paper-augmented agent accessed techniques published after its training cutoff (AdaGC, Feb 2025) and surfaced techniques it may have seen during training but didn't retrieve unprompted (sqrt scaling rule, 2022). This was deliberately tested on TinyStories — arguably the most well-explored small-scale setting in ML — to make the comparison harder. The effect would likely be larger on less-explored problems. Limitations: Single run per condition. The model is tiny (7M params). Some of the improvement may come from the agent spending more time reasoning about each technique rather than the paper content itself. More controlled ablations needed. I built the paper search MCP server (Paper Lantern) for this experiment. Free to try: https://code.paperlantern.ai Full writeup with methodology, all 15 paper citations, and appendices: https://www.paperlantern.ai/blog/auto-research-case-study Would be curious to see this replicated at larger scale or on different domains. submitted by /u/kalpitdixit [link] [comments]
Real-World Web Tasks Trip Up Even the Best AI Agents
We introduce ClawBench, a benchmark that evaluates AI browser agents on 153 real-world everyday tasks across 144 live websites. Unlike synthetic benchmarks, ClawBench tests agents on actual production platforms. Key findings: The best model (Claude Sonnet 4.6) achieves only 33.3% success rate GLM-5 (Zhipu AI) comes second at 24.2% — surprisingly strong for a text-only model Finance and Academic tasks are easier (50% for the best model); Travel and Dev tasks are much harder No model exceeds 50% in any category — there's a long way to go What makes ClawBench different: Tasks on real live websites, not sandboxed environments 5 layers of behavioral data: session replay, screenshots, HTTP traffic, agent reasoning traces, browser actions Request interceptor blocks the final HTTP request before irreversible actions (payments, bookings), enabling safe evaluation Human ground-truth for every task Agentic evaluator with step-level traceable diagnostics Resources: Paper: https://arxiv.org/abs/2604.08523 Website (interactive leaderboard + trace viewer): https://claw-bench.com Dataset: https://huggingface.co/datasets/NAIL-Group/ClawBench GitHub: https://github.com/reacher-z/ClawBench PyPI: pip install clawbench-eval Happy to answer any questions! We're actively looking for feedback on task selection and evaluation methodology. [R] Research submitted by /u/Extreme_Play_8554 [link] [comments]
An autonomous AI researcher that refines itself on your data
Inspired by Andrej Karpathy's AutoResearch, I built a system where Claude Code acts as an autonomous ML researcher on tabular binary classification tasks (churn, conversion, etc.). You give it a dataset. It loops forever: analyze data, form hypothesis, edit code, run experiment, evaluate with expanding time windows (train on past, predict future - no leakage), keep or revert via git. It edits only 3 files - feature engineering, model hyperparams, and analysis code. Everything else is locked down. Edit: To clarify based on some comments, I am using this to solve the problem of finding new signals to add to the model, not trying to overfit a limited dataset. -end Edit- Key design decisions: Introducing an analysis loop in addition to the experiment loop, this allow for better reflection and experimentation. Optimize for experiment throughput with a bunch of decisions: Use LightGBM as default model, limit feature count and tree count, locking down training run until it finishes. Constrained editing surface: only 3 files + logs. No infrastructure changes, no package installs. Without this, the agent will eventually try to modify the evaluation code to "improve" its score. Docker sandbox - the agent runs with full shell access (--dangerously-skip-permissions). Container keeps it contained. Expanding time windows over k-fold - mean score across multiple temporal train/test splits. Forced logging - every experiment gets a LOG.md entry (hypothesis, result, takeaway). Significant insights go to LEARNING.md. You can read the agent's reasoning after the fact. Analysis primitives built-in - univariate AUC, correlation pairs, null rates, feature importance, error analysis. The agent writes analysis code using these to save time, they also serve as initial suggestions for the first few analyses. What I learned building this: Air-tight evaluation is the essential for real improvement - this lesson hit me twice: Earlier version didn't constraint which file the agent could edit, it eventually changed the evaluation code to make "improvement" easier for itself. K-fold validation was originally employed, the agent found improvements that are actually data leakage and didn't hold out-of-time. After a painful manual inspection, I switched over to expanding time windows. Do everything to protect experiment throughput - this lesson also hit twice: Initially, I let the model run wild and was not very impressed when it barely run 20 experiments overnight. Turns out, the agent engineered thousands of new features that slowed down training and crash some runs due to RAM limit. I added the feature count limit and tree count limit to make sure training time is reasonable. Despite that, the agent still manage to crash/slow down training runs by putting many of them into background process at the same time. -> Locking mechanism was implemented to prevent 2 experiments being run at the same time. After this, the rate of progress increased to hundreds of runs per day. Persistent memory is important: Without forced logging, the agent would repeat experiments it already tried. The LOG.md and LEARNING.md system gives it memory across iterations. The code open source (sanitized version): https://github.com/trantrikien239/autoresearch-tabularOf course it is done with Claude Code, but it has improved so much after rounds of iterations, including manual edits, so I think it's worth sharing. submitted by /u/Pancake502 [link] [comments]