mlsanityData to 7 Oct 2026

MLGym: A New Framework and Benchmark for Advancing AI Research Agents

arXiv 2502.14499 ↗First posted 20 Feb 2025cs.CL cs.AI cs.LGcited by 111

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. This is the first Gym environment for machine learning (ML) tasks, enabling research on reinforcement learning (RL) algorithms for training such agents. MLGym-bench consists of 13 diverse and open-ended AI research tasks from diverse domains such as computer vision, natural language processing, reinforcement learning, and game theory. Solving these tasks requires real-world AI research skills such as generating new ideas and hypotheses, creating and processing data, implementing ML methods, training models, running experiments, analyzing the results, and iterating through this process to improve on a given task. We evaluate a number of frontier large language models (LLMs) on our benchmarks such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5 Pro. Our MLGym framework makes it easy to add new tasks, integrate and evaluate models or agents, generate synthetic data at scale, as well as develop new learning algorithms for training agents on AI research tasks. We find that current frontier models can improve on the given baselines, usually by finding better hyperparameters, but do not generate novel hypotheses, algorithms, architectures, or substantial improvements. We open-source our framework and benchmark to facilitate future research in advancing the AI research capabilities of LLM agents.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 100 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

53 of 100 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts22 Nov
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct
  3. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery7 Oct
  4. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories11 Sep
  5. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
  6. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?22 Jul
  7. 2024Data Interpreter: An LLM Agent For Data Science28 Feb
  8. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  10. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
31 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  4. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  5. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  6. 2024AutoCodeRover: Autonomous Program Improvementcited by 272
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  9. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  10. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
  11. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  12. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  13. 2018Learning a SAT Solver from Single-Bit Supervisioncited by 198
  14. 2024Tree Search for Language Model Agentscited by 148
  15. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  16. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  17. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
  18. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  19. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 89
  20. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  21. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  22. 2024AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitionscited by 70
  23. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  24. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  25. 2019Best Practices for Scientific Research on Neural Architecture Searchcited by 53
  26. 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 39
  27. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  28. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  29. 2017Pitfalls and Best Practices in Algorithm Configurationcited by 25
  30. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 22
  31. 2022AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 44; 2026: 67.

111 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 65
  6. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  7. 2025LLM Agents for Education: Advances and Applicationscited by 59
  8. 2025A Self-Improving Coding Agentcited by 58
  9. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 48
  10. 2025El Agente: An Autonomous Agent for Quantum Chemistrycited by 43
  11. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  12. 2025ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Taskscited by 41
38 more
  1. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  2. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  3. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27
  4. 2025Sparks of Science: Hypothesis Generation Using Structured Paper Datacited by 22
  5. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  6. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  7. 2026Towards Execution-Grounded Automated AI Researchcited by 19
  8. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 18
  9. 2025Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Taskscited by 17
  10. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  11. 2025Meta-RL Induces Exploration in Language Agentscited by 17
  12. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 16
  13. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  14. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 15
  15. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  16. 2026Autonomous Agents on Blockchains: Standards, Execution Models, and Trust Boundariescited by 12
  17. 2026AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?cited by 11
  18. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  19. 2025DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidencecited by 10
  20. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  21. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  22. 2026RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agentscited by 10
  23. 2026AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvementcited by 10
  24. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  25. 2025MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipelinecited by 9
  26. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 9
  27. 2026NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?cited by 9
  28. 2025On the Failure of Latent State Persistence in Large Language Modelscited by 8
  29. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 8
  30. 2025AI & Human Co-Improvement for Safer Co-Superintelligencecited by 8
  31. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  32. 2026Can AI agents conduct open-ended AI research? Early evidence from two case studiescited by 7
  33. 2026Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineeringcited by 7
  34. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  35. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 6
  36. 2026MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibilitycited by 6
  37. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  38. 2025LLM-First Search: Self-Guided Exploration of the Solution Spacecited by 5

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
  2. 2025FML-bench: Benchmarking Machine Learning Agents for Scientific Research
  3. 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
  4. 2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering
  5. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
  6. 2025Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization
  7. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
  8. 2025TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents