mlsanityData to 7 Oct 2026

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

arXiv 2410.07095 ↗First posted 9 Oct 2024cs.CLcited by 429

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents.

Key facts. Cited by 429 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 50 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

16 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?12 Sep
  2. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct

Other references · 14

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2024AI Agents That Mattercited by 199
  10. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  11. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  12. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
2 more
  1. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agentscited by 27
  2. 2024Discovering and exploring cases of educational source code plagiarism with Doloscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 129; 2026: 289.

429 papers.

Sort:

By area:

  1. 2024OpenAI o1 System Cardcited by 1,220
  2. 2025Humanity's Last Examcited by 467
  3. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  7. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  8. 2024Frontier Models are Capable of In-context Schemingcited by 252
  9. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  10. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  11. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  12. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
38 more
  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025Survey on Evaluation of LLM-based Agentscited by 190
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  6. 2026Towards End-to-End Automation of AI Researchcited by 150
  7. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  8. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  9. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  10. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  11. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  12. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  13. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  14. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 74
  15. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  16. 2024AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitionscited by 70
  17. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  18. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 65
  19. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  20. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  21. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  22. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 57
  23. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 56
  24. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 52
  25. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 48
  26. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 45
  27. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 44
  28. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  29. 2026Code as Agent Harnesscited by 40
  30. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 38
  31. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  32. 2025A sketch of an AI control safety casecited by 36
  33. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
  34. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  35. 2025R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Sciencecited by 33
  36. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 32
  37. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  38. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 31

50 most cited of 429 All 429 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows
  2. 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
  3. 2026Efficient Benchmarking of AI Agents
  4. 2026Assembling Insights for Agentic Machine Learning Engineering Systems
  5. 2025Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset
  6. 2026How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
  7. 2026Benchmark Everything Everywhere All at Once
  8. 2026TML-Bench: Benchmark for Data Science Agents on Tabular ML Tasks