mlsanityData to 7 Oct 2026

MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?

arXiv 2504.09702 ↗First posted 13 Apr 2025cs.AIcited by 27

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike prior work, e.g., AI Scientist, which evaluates the end-to-end agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of proposing and implementing novel research methods and evaluates them with rigorous protocol and objective metrics. Our curated suite of 7 competition tasks reveals significant challenges for LLM agents. Even the best-performing tested agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human participant scores. Furthermore, our analysis reveals a misalignment between the LLM-judged innovation and actual performance on cutting-edge ML research problems. MLRC-Bench is a dynamic benchmark, designed to grow with new ML competitions and encourage rigorous, objective evaluations of AI research capabilities. Our leaderboard and code are available at: https://huggingface.co/spaces/launch/MLRC_Bench

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 71 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

46 of 71 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search10 Apr
  2. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agents20 Feb
  3. 2025Agent Laboratory: Using LLM Agents as Research Assistants8 Jan
  4. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts22 Nov
  5. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents17 Oct
  6. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models9 Oct
  7. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct
  8. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?12 Sep
  9. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
  10. 2024MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents26 Aug
  11. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Models1 Jul
  12. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct

Other references · 34

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
22 more
  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  4. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  5. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  6. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
  7. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  8. 2024AI Agents That Mattercited by 199
  9. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  10. 2024Tree Search for Language Model Agentscited by 148
  11. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  12. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 84
  13. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  14. 2024Are we making progress in unlearning? Findings from the first NeurIPS unlearning competitioncited by 48
  15. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  16. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  17. 2024AAAR-1.0: Assessing AI's Potential to Assist Researchcited by 24
  18. 2024SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learningcited by 23
  19. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  20. 2024Perception Test 2024: Challenge Summary and a Novel Hour-Long VideoQA Benchmarkcited by 5
  21. 2022NeurIPS'22 Cross-Domain MetaDL competition: Design and baseline resultscited by 3
  22. 2025SRLCG: Self-Rectified Large-Scale Code Generation with Multidimensional Chain-of-Thought and Dynamic Backtrackingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 25.

27 papers.

Sort:

By area:

  1. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  2. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  3. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 15
  4. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 13
  5. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  6. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  7. 2026AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvementcited by 10
  8. 2026How Far Are We From True Auto-Research?cited by 8
  9. 2026Can AI agents conduct open-ended AI research? Early evidence from two case studiescited by 7
  10. 2026Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineeringcited by 7
  11. 2026Learning to Ideate for Machine Learning Engineering Agentscited by 6
  12. 2026AdaMEM: Test-Time Adaptive Memory for Language Agentscited by 4
15 more
  1. 2026PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Reviewcited by 3
  2. 2026SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?cited by 3
  3. 2026ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environmentscited by 1
  4. 2026Mimosa Framework: Toward Evolving Multi-Agent Systems for Scientific Researchcited by 1
  5. 2026SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystemscited by 1
  6. 2026Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarkscited by 1
  7. 2026Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Designcited by 1
  8. 2026When Does AI for PDEs Yield Scientific Evidence?cited by 1
  9. 2026Can LLM Agents Discover? Evaluating Creativity on ML Engineering Taskscited by 1
  10. 2026QuantumMind: Constraint-Grounded Agentic Reasoning for Speedup Analysis in Quantum Computingcited by 0
  11. 2026When AI Designs AI: Innovation or Imitation?cited by 0
  12. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  13. 2026Reward-rate Policy Gradient for Efficient Machine Learning Engineering Agentscited by 0
  14. 2026OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciencescited by 0
  15. 2026How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
  2. 2025FML-bench: Benchmarking Machine Learning Agents for Scientific Research
  3. 2025Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization
  4. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench
  5. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
  6. 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
  7. 2025ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows
  8. 2026Benchmark Everything Everywhere All at Once