mlsanityData to 7 Oct 2026

Agent Laboratory: Using LLM Agents as Research Assistants

arXiv 2501.04227 ↗First posted 8 Jan 2025cs.HC cs.AI cs.CLcited by 405

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Historically, scientific discovery has been a lengthy and costly process, demanding substantial time and resources from initial conception to final results. To accelerate scientific discovery, reduce research costs, and improve research quality, we introduce Agent Laboratory, an autonomous LLM-based framework capable of completing the entire research process. This framework accepts a human-provided research idea and progresses through three stages--literature review, experimentation, and report writing to produce comprehensive research outputs, including a code repository and a research report, while enabling users to provide feedback and guidance at each stage. We deploy Agent Laboratory with various state-of-the-art LLMs and invite multiple researchers to assess its quality by participating in a survey, providing human feedback to guide the research process, and then evaluate the final paper. We found that: (1) Agent Laboratory driven by o1-preview generates the best research outcomes; (2) The generated machine learning code is able to achieve state-of-the-art performance compared to existing methods; (3) Human involvement, providing feedback at each stage, significantly improves the overall quality of research; (4) Agent Laboratory significantly reduces research expenses, achieving an 84% decrease compared to previous autonomous research methods. We hope Agent Laboratory enables researchers to allocate more effort toward creative ideation rather than low-level coding and writing, ultimately accelerating scientific discovery.

Key facts. Cited by 405 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 67 of its 114 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

67 of 114 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents17 Oct
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct
  3. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery7 Oct
  4. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?12 Sep
  5. 2024SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning9 Sep
  6. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
  7. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science19 Aug
  8. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Models1 Jul
  9. 2024ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research Agents13 Jun
  10. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning27 Feb
  11. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  12. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks10 Jan
2 more
  1. 2024MARG: Multi-Agent Review Generation for Scientific Papers8 Jan
  2. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Research8 Dec

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
38 more
  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  4. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  5. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2022Large Language Models Can Self-Improvecited by 656
  8. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  9. 2024Automated Design of Agentic Systemscited by 360
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  12. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  13. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  14. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  15. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  16. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  17. 2024LLM Agents can Autonomously Hack Websitescited by 108
  18. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  19. 2024Towards Conversational Diagnostic AIcited by 79
  20. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agentscited by 76
  21. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  22. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  23. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 56
  24. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  25. 2024Autonomous LLM-driven research from data to human-verifiable research paperscited by 50
  26. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  27. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
  28. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 45
  29. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 38
  30. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  31. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  32. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  33. 2024How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experimentcited by 22
  34. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 22
  35. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21
  36. 2024MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflowscited by 19
  37. 2024Automating Exploratory Proteomics Research via Language Modelscited by 11
  38. 2024BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Sciencecited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 149; 2026: 256.

405 papers.

Sort:

By area:

  1. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 375
  2. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  3. 2025Video models are zero-shot learners and reasonerscited by 208
  4. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  7. 2025Agentic Large Language Models, a surveycited by 105
  8. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  9. 2025How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behaviorcited by 95
  10. 2025AI-Researcher: Autonomous Scientific Innovationcited by 88
  11. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  12. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
38 more
  1. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 74
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  3. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  4. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  5. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  6. 2025Deep Research: A Survey of Autonomous Research Agentscited by 63
  7. 2025LLM Agents for Education: Advances and Applicationscited by 59
  8. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  9. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  10. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 56
  11. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 52
  12. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  13. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 49
  14. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  15. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  16. 2025LLM Agents Making Agent Toolscited by 45
  17. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  18. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  19. 2025General Agentic Memory Via Deep Researchcited by 42
  20. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  21. 2026EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discoverycited by 41
  22. 2025TxGemma: Efficient and Agentic LLMs for Therapeuticscited by 40
  23. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  24. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  25. 2026Code as Agent Harnesscited by 40
  26. 2025Self-Supervised Prompt Optimizationcited by 37
  27. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  28. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  29. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  30. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  31. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 31
  32. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
  33. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 31
  34. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  35. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 29
  36. 2025FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spacescited by 28
  37. 2025AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprisecited by 28
  38. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27

50 most cited of 405 All 405 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Vision for Auto Research with LLM Agents
  2. 2026Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts
  3. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research
  4. 2026FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
  5. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  6. 2025Agent4S: The Transformation of Research Paradigms from the Perspective of Large Language Models
  7. 2025ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies
  8. 2024Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System