mlsanityData to 7 Oct 2026

Training AI Scientists to Replicate Research

arXiv 2608.13331 ↗First posted 13 Aug 2026cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we develop Replica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-train Faraday, a 27B-parameter "AI Scientist" agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 101 of its 181 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

101 of 181 references.

References · 101

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  12. 2023Large Language Models as Optimizerscited by 830
38 more
  1. 2022A Generalist Agentcited by 802
  2. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  3. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 762
  4. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  6. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 467
  9. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  10. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  11. 2017Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equationscited by 362
  12. 2024Automated Design of Agentic Systemscited by 360
  13. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  14. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  15. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  16. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  17. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  18. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  19. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  20. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  21. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
  22. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  23. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  24. 2025ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolutioncited by 146
  25. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  26. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  27. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  28. 2022Evolution through Large Modelscited by 131
  29. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  30. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  31. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  32. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  33. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  34. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  35. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  36. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
  37. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  38. 2025AI-Researcher: Autonomous Scientific Innovationcited by 88

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Recursive self-improvement of AI research agentscited by 3
  2. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  3. 2026EurekaBench: Measuring Agentic Ability to Discover New Scientific Insightscited by 0
  4. 2026Agents Are Systems, Not Models: Rethinking Agentic Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
  2. 2026Coding-agents can replicate scientific machine learning papers
  3. 2026Towards Automating Scientific Review with Google's Paper Assistant Tool
  4. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
  5. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap
  6. 2025ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
  7. 2026Towards End-to-End Automation of AI Research
  8. 2026A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline