mlsanityData to 14 Oct 2026

ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code

arXiv 2506.02314 ↗First posted 2 Jun 2025cs.AI cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown promise in transforming machine learning research, yet their capability to faithfully implement novel ideas from recent research papers-ideas unseen during pretraining-remains unclear. We introduce ResearchCodeBench, a benchmark of 212 coding challenges that evaluates LLMs' ability to translate cutting-edge ML contributions from top 2024-2025 research papers into executable code. We assessed 30+ proprietary and open-source LLMs, finding that even the best models correctly implement less than 40% of the code. We find Gemini-2.5-Pro-Preview to perform best at 37.3% success rate, with O3 (High) and O4-mini (High) following behind at 32.3% and 30.8% respectively. We present empirical findings on performance comparison, contamination, and error patterns. By providing a rigorous and community-driven evaluation platform, ResearchCodeBench enables continuous understanding and advancement of LLM-driven innovation in research code generation.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

36 of 50 references.

References · 36

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  3. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  5. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  6. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  7. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  8. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  9. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  10. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  11. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 225
  12. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
24 more
  1. 2024The Road Less Scheduledcited by 163
  2. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  3. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  4. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  5. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  6. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 118
  7. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  8. 2025Transformers without Normalizationcited by 70
  9. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 69
  10. 2024TabDiff: a Mixed-type Diffusion Model for Tabular Data Generationcited by 63
  11. 2025Your ViT is Secretly an Image Segmentation Modelcited by 63
  12. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  13. 2025Fractal Generative Modelscited by 46
  14. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 44
  15. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  16. 2025"Principal Components" Enable A New Language of Imagescited by 27
  17. 2025Data Unlearning in Diffusion Modelscited by 18
  18. 2025Gaussian Mixture Flow Matching Modelscited by 17
  19. 2024Attention as a Hypernetworkcited by 13
  20. 2024Advantage Alignment Algorithmscited by 7
  21. 2025Optimal Stepsize for Diffusion Samplingcited by 7
  22. 2024On Conformal Isometry of Grid Cells: Learning Distance-Preserving Position Embeddingcited by 6
  23. 2024Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysiscited by 5
  24. 2024Second-Order Min-Max Optimization with Lazy Hessianscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 31.

36 papers.

Sort:

By area:

  1. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  2. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 25
  3. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 22
  4. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 19
  5. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 18
  6. 2026The Last Human-Written Paper: Agent-Native Research Artifactscited by 16
  7. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  8. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  9. 2026ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciencescited by 9
  10. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  11. 2026Training AI Scientists to Replicate Researchcited by 9
  12. 2026PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generationcited by 8
24 more
  1. 2026MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibilitycited by 8
  2. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agentscited by 6
  3. 2026SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?cited by 6
  4. 2026Read the Paper, Write the Code: Agentic Reproduction of Social-Science Resultscited by 5
  5. 2026The Implementation Lottery: Auditing Idea Reliability in Automated Researchcited by 5
  6. 2026SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?cited by 3
  7. 2026Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecyclecited by 3
  8. 2026Benchmarking PhD-Level Coding in 3D Geometric Computer Visioncited by 2
  9. 2026FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verificationcited by 2
  10. 2026From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligencecited by 2
  11. 2026MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmarkcited by 1
  12. 2026REVERE: Reflective Evolving Research Engineercited by 1
  13. 2026AlphaEval: Evaluating Agents in Productioncited by 1
  14. 2026Coding-agents can replicate scientific machine learning paperscited by 1
  15. 2025CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agentscited by 0
  16. 2026AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositoriescited by 0
  17. 2026Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solvercited by 0
  18. 2026Which Algorithm Specification Formats Help Language Models Implement Machine Learning Algorithms?cited by 0
  19. 2026Can LLMs Build a MaxSAT Solver from Papers? The CoreForge Experiencecited by 0
  20. 2026From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesiscited by 0
  21. 2026Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluationcited by 0
  22. 2026PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progresscited by 0
  23. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  24. 2026ReproBAIT: Can coding agents build their own biological AI tools?cited by 0