mlsanityData to 14 Oct 2026

PaperBench: Evaluating AI's Ability to Replicate AI Research

arXiv 2504.01848 ↗First posted 2 Apr 2025cs.AI cs.CLcited by 321

Not on the map. Why?

Cited by 145 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria. In total, PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed with the author(s) of each ICML paper for accuracy and realism. To enable scalable evaluation, we also develop an LLM-based judge to automatically grade replication attempts against rubrics, and assess our judge's performance by creating a separate benchmark for judges. We evaluate several frontier models on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, achieves an average replication score of 21.0%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench, finding that models do not yet outperform the human baseline. We open-source our code (https://github.com/openai/preparedness) to facilitate future research in understanding the AI engineering capabilities of AI agents.

Key facts. Cited by 321 papers in the map (data through 14 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

35 of 55 references.

References · 35

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  4. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  6. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  7. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  8. 2023GPTScore: Evaluate as You Desirecited by 361
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  10. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  11. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  12. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
23 more
  1. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  2. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  3. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  4. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  5. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  6. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 113
  7. 2023Stochastic interpolants with data-dependent couplingscited by 100
  8. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  9. 2023Stay on topic with Classifier-Free Guidancecited by 91
  10. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  11. 2024All-in-one simulation-based inferencecited by 74
  12. 2022Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Modelscited by 50
  13. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  14. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  15. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  16. 2024Batch and match: black-box variational inference with a score-based divergencecited by 22
  17. 2024SAPG: Split and Aggregate Policy Gradientscited by 21
  18. 2024Sample-specific Masks for Visual Reprogramming-based Promptingcited by 18
  19. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  20. 2024LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomiescited by 12
  21. 2024BBox-Adapter: Lightweight Adapting for Black-Box Large Language Modelscited by 11
  22. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
  23. 2024RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 84; 2026: 237.

321 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2025Kimi K2: Open Agentic Intelligencecited by 567
  3. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  4. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  5. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  6. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  7. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 101
  8. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 76
  9. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  10. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
  11. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 70
  12. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 65
38 more
  1. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 64
  2. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 63
  3. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  4. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 47
  5. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  6. 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 39
  7. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 38
  8. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  9. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 36
  10. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 34
  11. 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Developmentcited by 34
  12. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 33
  13. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 32
  14. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  15. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  16. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  17. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  18. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 26
  19. 2025MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problemcited by 25
  20. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  21. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 25
  22. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 24
  23. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 24
  24. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  25. 2026AutoFigure: Generating and Refining Publication-Ready Scientific Illustrationscited by 23
  26. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 23
  27. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 23
  28. 2026Toward Generalist Autonomous Research via Hypothesis-Tree Refinementcited by 22
  29. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 21
  30. 2026Structured Scaling of AI Discovery Across Diverse Scientific Domainscited by 21
  31. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 20
  32. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  33. 2026In-Place Test-Time Trainingcited by 20
  34. 2025RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solvingcited by 19
  35. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 19
  36. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 19
  37. 2026ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidencecited by 19
  38. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 19

50 most cited of 321 All 321 (JSON)