# Coding-agents can replicate scientific machine learning papers

arXiv: https://arxiv.org/abs/2607.02134 · HTML: https://mlsanity.com/paper/2607.02134/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 22 of 22)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code](https://mlsanity.com/paper/2506.02314/) (2025)
- [LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research](https://mlsanity.com/paper/2506.17335/) (2025)
- [REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?](https://mlsanity.com/paper/2507.18901/) (2025)
- [ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies](https://mlsanity.com/paper/2504.20117/) (2025)
- [Terminologies for Reproducible Research](https://mlsanity.com/paper/1802.03311/) (2018)
- [Good Enough Practices in Scientific Computing](https://mlsanity.com/paper/1609.00037/) (2016)
- [Physics-informed Information Field Theory for Modeling Physical Systems with Uncertainty Quantification](https://mlsanity.com/paper/2301.07609/) (2023)
- [Can Coding Agents Reproduce Findings in Computational Materials Science?](https://mlsanity.com/paper/2605.00803/) (2026)
- [Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction](https://mlsanity.com/paper/2605.13950/) (2026)
- [AI Coding Agents Can Reproduce Social Science Findings](https://mlsanity.com/paper/2606.11447/) (2026)
- [Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results](https://mlsanity.com/paper/2604.21965/) (2026)
- [Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches](https://mlsanity.com/paper/2602.08561/) (2026)

## Built on by (most cited first, 1 of 1)

- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)

## Abstract (arXiv)

Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.