# NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

arXiv: https://arxiv.org/abs/2606.24530 · HTML: https://mlsanity.com/paper/2606.24530/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 34 of its 88 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 34)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [MLGym: A New Framework and Benchmark for Advancing AI Research Agents](https://mlsanity.com/paper/2502.14499/) (2025)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Equivariant 3D-Conditional Diffusion Models for Molecular Linker Design](https://mlsanity.com/paper/2210.05274/) (2022)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Learning to Discover at Test Time](https://mlsanity.com/paper/2601.16175/) (2026)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively](https://mlsanity.com/paper/2509.26603/) (2025)
- [EvoX: Meta-Evolution for Automated Discovery](https://mlsanity.com/paper/2602.23413/) (2026)
- [EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery](https://mlsanity.com/paper/2603.08127/) (2026)
- [MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering](https://mlsanity.com/paper/2505.07782/) (2025)
- [AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents](https://mlsanity.com/paper/2602.06855/) (2026)
- [LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research](https://mlsanity.com/paper/2506.17335/) (2025)
- [DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolution](https://mlsanity.com/paper/2602.02919/) (2026)
- [AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://mlsanity.com/paper/2604.05550/) (2026)
- [REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?](https://mlsanity.com/paper/2507.18901/) (2025)
- [Scientific Algorithm Discovery by Augmenting AlphaEvolve with Deep Research](https://mlsanity.com/paper/2510.06056/) (2025)
- [Position: Agentic Evolution is the Path to Evolving LLMs](https://mlsanity.com/paper/2602.00359/) (2026)
- [AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?](https://mlsanity.com/paper/2606.05080/) (2026)
- [From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking](https://mlsanity.com/paper/2506.19724/) (2025)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [How Far Are We From True Auto-Research?](https://mlsanity.com/paper/2605.19156/) (2026)
- [Can Coding Agents Reproduce Findings in Computational Materials Science?](https://mlsanity.com/paper/2605.00803/) (2026)
- [EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale](https://mlsanity.com/paper/2604.17406/) (2026)

## Built on by (most cited first, 10 of 10)

- [Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering](https://mlsanity.com/paper/2607.28568/) (2026)
- [How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks](https://mlsanity.com/paper/2608.14905/) (2026)
- [BrainPilot: Automating Brain Discovery with Agentic Research](https://mlsanity.com/paper/2607.15079/) (2026)
- [FrontierChallenge: Evaluating Scientific Workflow Completion](https://mlsanity.com/paper/2608.24979/) (2026)
- [AIBuildAI-2.5: Efficient Autonomous AI Model Development Through LLM-Guided Tree Search](https://mlsanity.com/paper/2609.25047/) (2026)
- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)
- [On the Behavioral Traits of LLM Agents](https://mlsanity.com/paper/2609.32776/) (2026)
- [EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights](https://mlsanity.com/paper/2610.00492/) (2026)
- [Agents Are Systems, Not Models: Rethinking Agentic Evaluation](https://mlsanity.com/paper/2610.01618/) (2026)
- [CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery](https://mlsanity.com/paper/2610.04851/) (2026)

## Abstract (arXiv)

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench