mlsanityData to 14 Oct 2026

MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

arXiv 2310.03302 ↗First posted 5 Oct 2023cs.LG cs.AIcited by 366

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

A central aspect of machine learning research is experimentation, the process of designing and running experiments, analyzing the results, and iterating towards some positive outcome (e.g., improving accuracy). Could agents driven by powerful language models perform machine learning experimentation effectively? To answer this question, we introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We then construct an agent that can perform ML experimentation based on ReAct framework. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate. It can build compelling ML models over many tasks in MLAgentBench with 37.5% average success rate. Our agents also display highly interpretable plans and actions. However, the success rates vary considerably; they span from 100% on well-established older datasets to as low as 0% on recent Kaggle challenges created potentially after the underlying LM was trained. Finally, we identify several key challenges for LM-based agents such as long-term planning and reducing hallucination. Our code is released at https://github.com/snap-stanford/MLAgentBench.

Key facts. Cited by 366 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 52 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

21 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul

Other references · 20

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2019AutoML: A Survey of the State-of-the-Artcited by 390
8 more
  1. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  2. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
  3. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  4. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  5. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  6. 2017"Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Modelscited by 55
  7. 2023A Survey on Graph Diffusion Models: Generative AI in Science for Molecule, Protein and Materialcited by 35
  8. 2023Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systemscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 34; 2025: 116; 2026: 214.

366 papers.

Sort:

By area:

  1. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  2. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  3. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  4. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 226
  5. 2024AI Agents That Mattercited by 219
  6. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  7. 2026Towards End-to-End Automation of AI Researchcited by 159
  8. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  9. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  10. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  11. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  12. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
38 more
  1. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  2. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  3. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 97
  4. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  5. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 92
  6. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 77
  7. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 77
  8. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  9. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 71
  10. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 70
  11. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 65
  12. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
  13. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  14. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 64
  15. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 61
  16. 2025LLM Agents for Education: Advances and Applicationscited by 59
  17. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 58
  18. 2024Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideascited by 57
  19. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 55
  20. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 53
  21. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  22. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  23. 2025SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writingcited by 49
  24. 2026Code as Agent Harnesscited by 47
  25. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 46
  26. 2024DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent Collaborationcited by 44
  27. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 43
  28. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  29. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 41
  30. 2025R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Sciencecited by 41
  31. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  32. 2023LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystemcited by 38
  33. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 37
  34. 2024ResearchTown: Simulator of Human Research Communitycited by 37
  35. 2025HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generationcited by 37
  36. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 36
  37. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 36
  38. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 32

50 most cited of 366 All 366 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  2. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research
  3. 2025AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
  4. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents
  5. 2025APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
  6. 2024MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents
  7. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Research
  8. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks