When LLMs consult research papers, automated experiments gain a 3.2% edge.

In a controlled experiment, a Claude Code agent demonstrated a 3.2% improvement in results by accessing over 2 million computer science papers during automated hyperparameter searches. Utilizing Karpathy's autoresearch…

March 28, 20263 min readMachine Learning

When LLMs consult research papers, automated experiments gain a 3.2% edge. — [R] Controlled experiment: giving an LLM agent access to CS papers during automated hyperparameter search improves results by 3.2%

This experiment is the most practical argument we've seen for giving AI agents live access to research. A 3.2% improvement on a problem as well-trodden as TinyStories is not a fluke; it is a signal that the ceiling for automated experimentation is far higher than we assume. The agent without papers was limited to techniques already memorized in its weights. The paper-augmented agent found AdaGC, a method published after its training cutoff, and correctly applied the sqrt batch scaling rule on its first try. That is the difference between an agent that guesses and an agent that reads.

The practical takeaway for anyone running automated experiments is clear. Your LLM agent has a wall of knowledge it cannot see. It may have encountered a technique during training, but unless you prompt it to search, that knowledge stays latent. The paper-augmented agent didn't just find more techniques; it found the *right* ones faster. When the agent without papers halved the batch size and diverged because it didn't adjust the learning rate, the paper-augmented agent retrieved the scaling rule and applied it correctly. That saved time, compute, and the frustration of a failed run.

We also appreciate the honesty in the limitations. Some of the gain may come from the agent spending more time reasoning about each technique, not just from the paper content. That is still a win. Forcing the agent to search, read, and synthesize introduces a deliberation step that improves outcomes regardless of the source. The fact that DyT and SeeDNorm were tried and reverted shows the agent was not blindly applying every paper it found. It tested, observed, and discarded. That is actual scientific reasoning.

The next step is to replicate this at larger scale and on less-explored problems. TinyStories was a deliberate stress test to make the comparison harder. If the gap widens on novel domains, we will see a fundamental shift in how automated research works. For now, the message is simple: if you are running automated experiments without giving your agent access to the literature, you are leaving a measurable advantage on the table. Build the search layer. The 3.2% edge is just the start.

From Machine Learning

Ran a controlled experiment measuring whether LLM coding agents benefit from access to research literature during automated experimentation.
Two identical runs using Karpathy's autoresearch framework. Claude Code agent optimizing a ~7M param GPT-2 on TinyStories. M4 Pro, 100 experiments each, same seed config. Only variable — one agent had access to an MCP server that does full-text search over 2M+ CS papers and returns synthesized methods with citations.

Read the original at Machine Learning

Briefing

Source: Machine Learning
Published: March 28, 2026

An autonomous AI researcher that refines itself on your data
Inspired by Andrej Karpathy's AutoResearch, I built a system where Claude Code acts as an autonomous ML researcher on tabular binary classification tasks (churn, conversion, etc.). You give it a dataset. It loops forever: analyze data, form hypothesis, edit code, run experiment, evaluate with expanding time windows (train on past, predict future - no leakage), keep or revert via git. It edits only 3 files - feature engineering, model hyperparams, and analysis code. Everything else is locked down. Edit: To clarify based on some comments, I am using this to solve the problem of finding new signals to add to the model, not trying to overfit a limited dataset. -end Edit- Key design decisions: Introducing an analysis loop in addition to the experiment loop, this allow for better reflection and experimentation. Optimize for experiment throughput with a bunch of decisions: Use LightGBM as default model, limit feature count and tree count, locking down training run until it finishes. Constrained editing surface: only 3 files + logs. No infrastructure changes, no package installs. Without this, the agent will eventually try to modify the evaluation code to "improve" its score. Docker sandbox - the agent runs with full shell access (--dangerously-skip-permissions). Container keeps it contained. Expanding time windows over k-fold - mean score across multiple temporal train/test splits. Forced logging - every experiment gets a LOG.md entry (hypothesis, result, takeaway). Significant insights go to LEARNING.md. You can read the agent's reasoning after the fact. Analysis primitives built-in - univariate AUC, correlation pairs, null rates, feature importance, error analysis. The agent writes analysis code using these to save time, they also serve as initial suggestions for the first few analyses. What I learned building this: Air-tight evaluation is the essential for real improvement - this lesson hit me twice: Earlier version didn't constraint which file the agent could edit, it eventually changed the evaluation code to make "improvement" easier for itself. K-fold validation was originally employed, the agent found improvements that are actually data leakage and didn't hold out-of-time. After a painful manual inspection, I switched over to expanding time windows. Do everything to protect experiment throughput - this lesson also hit twice: Initially, I let the model run wild and was not very impressed when it barely run 20 experiments overnight. Turns out, the agent engineered thousands of new features that slowed down training and crash some runs due to RAM limit. I added the feature count limit and tree count limit to make sure training time is reasonable. Despite that, the agent still manage to crash/slow down training runs by putting many of them into background process at the same time. -> Locking mechanism was implemented to prevent 2 experiments being run at the same time. After this, the rate of progress increased to hundreds of runs per day. Persistent memory is important: Without forced logging, the agent would repeat experiments it already tried. The LOG.md and LEARNING.md system gives it memory across iterations. The code open source (sanitized version): https://github.com/trantrikien239/autoresearch-tabularOf course it is done with Claude Code, but it has improved so much after rounds of iterations, including manual edits, so I think it's worth sharing. submitted by /u/Pancake502 [link] [comments]
Measure your coding agent against a new open-source benchmark suite.
Sharing an open-source benchmark suite (paper-lantern-challenges) that measures coding-agent performance with vs without retrieval-augmented technique selection across 9 everyday software tasks. Disclosure: I'm the author of the retrieval system under test (paperlantern.ai/code); the artifact being shared here is the benchmark suite itself, not the product. Every prompt, agent code path, and prediction file is in the repo and reproducible. Setup. Same coding agent (Claude Opus 4.6 as the planner, Gemini Flash 3 as the task model), same input data, same evaluation scripts across all 9 tasks: test generation (mutation score), text-to-SQL (execution accuracy), PDF extraction, contract extraction, PR review, text classification, few-shot prompt selection, LLM routing, summarization evaluation. Independent variable: whether the agent could call a retrieval tool over CS literature before writing its solution. One pass per task, no retries, no manual filtering of outputs. Task selection. Tasks were chosen to span the everyday-engineering surface a coding agent actually faces, not specialized ML scenarios. Selection criteria: (1) unambiguous quantitative metric, (2) baseline performance well below ceiling, (3) standard datasets where they exist, (4) eval reproducible on a free Gemini API key in roughly 10 minutes per task. Eval methodology. Each task uses its task-standard quantitative metric (mutation score for test_generation, execution accuracy for text_to_sql, F1 on labeled spans for the extraction tasks, weighted F1 for classification, etc.). Full per-task scripts and dataset choices are in the repo - one directory per task, evaluate.py as the entry point, README.md per task documenting methodology and dataset. Retrieval setup. The "with retrieval" agent has access to three tool calls: explore_approaches(problem) returns ranked candidate techniques from the literature, deep_dive(technique) returns implementation steps and known failure modes for a chosen technique, compare_approaches(candidates) is for side-by-side when multiple options look viable. The agent decides when and how often to call them. Latency is roughly 20s per call; results cache across sessions. The baseline agent has none of these tools, otherwise identical scaffolding. Comparability. Both agents share the same task-specific user prompt; the only system-prompt difference is the retrieval agent's tool-call grammar. Predictions and per-task prompts are diffable in the repo (baseline/ and with_pl/ subdirectories per task). Results. Task Baseline With retrieval Delta extraction_contracts 0.444 0.764 +0.320 extraction_schemas 0.318 0.572 +0.254 test_generation 0.625 0.870 +0.245 classification 0.505 0.666 +0.161 few_shot 0.193 0.324 +0.131 code_review 0.351 0.395 +0.044 text_to_sql 0.650 0.690 +0.040 routing 0.744 0.761 +0.017 summeval 0.623 0.633 +0.010 The test-generation delta came from the agent discovering mutation-aware prompting - the techniques are MuTAP and MUTGEN - which enumerate every AST-level mutation of the target and require one test per mutation. Baseline wrote generic tests from pretrain priors. The contract extraction delta came from BEAVER (section-level relevance scoring) and PAVE (post-extraction validation), both 2026 techniques that post-date the agent's training. 10 of the 15 most-cited sources across the experiments were published in 2025 or later, which is the conservative argument for why retrieval matters: the agent could not have reached these techniques from parametric memory. Failure modes. Self-refinement hurt text-to-SQL (the agent second-guessed correct queries after reading work on SQL ambiguity). Two suggested techniques (DyT, SeeDNorm) were architecture-incompatible in the autoresearch experiment and got discarded. Retrieval surfaces better options, not guaranteed wins. Reproducibility. Every prompt, every line of agent code, every prediction file, every eval script is in the repo. Each task directory has a README documenting methodology and an approach.md showing exactly what the retrieval surfaced and which technique the agent chose. Repo: https://github.com/paperlantern-ai/paper-lantern-challenges Writeup with detailed per-task discussion: https://www.paperlantern.ai/blog/coding-agent-benchmarks Happy to share additional design choices in comments. submitted by /u/kalpitdixit [link] [comments]
Making AI research accessible with an 18x cheaper, open-source pipeline
TL;DR: I built an open-source pipeline that runs Karpathy's autoresearch on SageMaker Spot instances — 25 autonomous ML experiments for $0.44 total (vs ~$24 on an H100). 4x parallel execution, 2.3x faster, 18x cheaper. Includes an 8-chapter vibe coding tutorial. GitHub The Problem Karpathy's autoresearch is brilliant — an AI agent modifies training code, runs 5-minute experiments, keeps improvements, and repeats overnight. But it assumes you have an H100 sitting around for 8 hours. Most of us don't. I wanted to know: can you get the same results on cheap cloud GPUs, paying only pennies per experiment? What I Built A parallel evolution pipeline on SageMaker Managed Spot Training: Each generation: N candidates generated → N SageMaker Spot jobs run simultaneously → best val_bpb selected → next generation HUGI pattern (Hurry Up and Get Idle): GPUs spin up for 5 minutes, terminate immediately. Zero idle cost. Works with any GPU: H100, L40S, A10G — auto-detects and falls back gracefully Architecture: diagram Results Original (H100, sequential) This project (L40S Spot, parallel) Cost for 83 experiments ~$24 (on-demand) / ~$7 (spot) ~$1.33 Wall clock ~8 hours ~3.5 hours GPU idle cost ~50% wasted $0 Experiments in parallel 1 4 My actual run: 25 experiments across 5 generations for $0.44 on L40S (ml.g7e.2xlarge Spot in us-east-1). The pipeline autonomously discovered that EMBEDDING_LR is the most sensitive parameter, improving val_bpb from 1.0656 → 1.0643 through conservative LR evolution. Architecture changes (deeper models, bigger batches) all failed in the 5-minute budget. Surprises Along the Way Some things I learned the hard way: Spot capacity varies 1-9 by region. Same instance type: score 1 in us-west-2 (stuck for 30+ min), score 9 in us-east-1 (allocated in 2 min). Always run aws ec2 get-spot-placement-scores before choosing a region. Flash Attention 3 doesn't work on L40S. Pre-compiled FA3 kernels only support Hopper (sm_90) and Ampere (sm_80/86). Ada Lovelace (sm_89) crashes at runtime. Had to add a PyTorch SDPA fallback — which halved MFU (20% vs 40%). DEVICE_BATCH_SIZE ≠ throughput. Doubled batch size from 64→128, used 2x VRAM... and val_bpb got WORSE. Turns out with fixed TOTAL_BATCH_SIZE, larger micro-batches just reduce gradient accumulation steps without processing more tokens. The real lever is TOTAL_BATCH_SIZE. Larger Spot instances can be cheaper. g7e.8xlarge ($0.93/hr) was cheaper than g7e.2xlarge ($1.82/hr) because of lower demand. Check price history for all sizes. Cheap GPU experiments transfer to expensive GPUs. Research confirms that architecture/optimizer rankings found on L40S ($0.04/experiment) transfer to H100 for production training. Absolute LR values need re-tuning, but "A beats B" conclusions are portable. The Vibe Coding Angle The entire project was built through conversational AI coding (Claude Code) in a single ~13-hour session. I documented the full journey as an 8-chapter vibe coding tutorial — from initial idea through infrastructure debugging to autonomous evolution results. Every chapter includes the actual prompts used, the failures encountered, and the cost at each step. Try It ```bash git clone https://github.com/roboco-io/serverless-autoresearch cd serverless-autoresearch cp config.yaml.example config.yaml Edit with your AWS credentials make setup # IAM role make prepare # Data → S3 make dry-run # Verify (free) make run # 10 gen × 4 pop = 40 experiments (~$0.70) ``` Links GitHub: https://github.com/roboco-io/serverless-autoresearch Tutorial: 8-chapter vibe coding tutorial Comparison Report: Original vs Serverless Spot Capacity Guide: How to find available Spot GPUs Key Insights: 12 battle-tested lessons What's your cheapest setup for running ML experiments? Anyone tried autoresearch on other cloud providers? Update: I wrote a full step-by-step tutorial documenting how this was built. If you want to learn by doing (not just read the code), I turned the entire build process into an 8-chapter hands-on tutorial: | Ch | What You'll Learn | |----|------------------| | 1 | How a single prompt + deep interview became the architecture | | 2 | 23 files generated in one session with parallel AI agents | | 3 | The region saga — Spot scores, quota wars, 3 region migrations | | 4 | First experiment: FA3 CUDA crash → SDPA fallback → $0.02 success | | 5 | The Batch Size Trap — why doubling BS made results WORSE | | 6 | 5 generations of autonomous evolution (what worked vs what failed) | | 7 | Turning lessons into a reusable Claude Code skill | | 8 | Final scorecard: 18x cheaper, 2.3x faster | Every chapter includes the actual prompt I used, what went wrong, and exact commands to reproduce it. Total cost to follow along: ~$0.70. The most educational part is probably Chapter 5 (The Batch Size Trap) — I learned that DEVICE_BATCH_SIZE ≠ throughput the hard way ($0.07 lesson). Start here: Chapter 1: The Idea submitted by /u/Consistent-Milk-6643 [link] [comments]