frontier models

How frontier models master the real-world grind of materials research

Seven frontier AI models, across twelve runs, scored between 97.8 and 100 on a PhD-level materials research test that included unit traps, hidden crystal structures, and a manuscript with planted errors. Every model…

3 min readMachine Learning
How frontier models master the real-world grind of materials research
Four frontier models scored 97.8–100 on a 7-part materials-science research test. How should we score long-horizon agent evals instead? [d]

The boundary between what machines can execute and what they can discover is narrowing fast, and the implications for research workflows are immediate. A materials science PhD student recently built a rigorous seven-part test of routine lab work, data cleaning against a protocol, physics calculations, manuscript critique, experimental design, and found that four frontier AI models scored between 97.8 and 100 out of 100, completing the tasks in 7 to 24 minutes. Every run caught the planted unit trap, passed the hidden structure checks, and identified all ten deliberate errors in a manuscript without a single false positive. Three runs even spotted a real mistake the test author hadn't planted. This isn't a lab curiosity; it is a signal that bounded, procedure-heavy research tasks have reached a saturation point for AI capability. For anyone who has spent an afternoon wrestling with a messy tensile data set or checking a collaborator's CIF file, the practical takeaway is straightforward: the grunt work that once consumed hours can now be delegated reliably.

This result lands alongside two related observations we have been tracking. Our piece on Let AI Polish Your Ideas, Not Dream Them Up examined how AI excels at refining and executing within defined constraints but struggles with original conceptual leaps. And the story of Physicl’s $10M bet bridges digital commerce and physical robot training highlighted the persistent difficulty of transferring simulated competence into unpredictable physical environments. The graphene test fits neatly between those two poles: the models mastered the known procedures, but the test author himself notes the gap lies in long-horizon, open-ended work, the kind where a researcher recovers from a wrong early assumption or pivots when an instrument breaks. That is precisely the frontier that remains genuinely human, and it is the right place to focus attention.

The test author is already building the next evaluation: a simulated-lab campaign where an agent gets a budget, instruments, and a target spec, with planted surprises of the kind real projects generate. He asks how to score trajectory separately from final output, and whether anyone has measured human baselines on agentic tasks of that length. Those are the questions that matter now, because the ceiling for bounded execution is established. The ceiling for adaptive, surprise-handling, recovery-from-error work is not. If your research group relies on people to clean data and check units, those tasks are already candidates for automation. But if your group depends on someone who can sense when the experiment is lying to them and change course, that skill is still the bottleneck. Watch how the simulated-lab results compare against the PhD student's own estimate of 11-16 hours of focused work and an expected score of 75-90. That delta is the real measure of where we are.

From Machine Learning

I work in graphene and 2D materials, and I built a test of the everyday technical work in a research group. It has seven parts: reducing a messy tensile data set against a lab SOP (with unit and outlier traps), writing a CIF density function checked on three hidden structures, four back-of-envelope physics calculations, critiquing a manuscript with 10 planted errors and 6 correct-but-suspicious decoys, false-premise and fabricated-paper questions with calibration scoring, an experimental design, and an abstract under constraints. All the data are synthetic, the answer keys are computed by code, and it's scored out of 100 across 8…

Read the original at Machine Learning