mlsanityData to 7 Oct 2026

Agentic Large Language Models, a survey

arXiv 2503.23037 ↗First posted 29 Mar 2025cs.AI cs.CL cs.LGcited by 105

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Background: There is great interest in agentic LLMs, large language models that act as agents. Objectives: We review the growing body of work in this area and provide a research agenda. Methods: Agentic LLMs are LLMs that (1) reason, (2) act, and (3) interact. We organize the literature according to these three categories. Results: The research in the first category focuses on reasoning, reflection, and retrieval, aiming to improve decision making; the second category focuses on action models, robots, and tools, aiming for agents that act as useful assistants; the third category focuses on multi-agent systems, aiming for collaborative task solving and simulating interaction to study emergent social behavior. We find that works mutually benefit from results in other categories: retrieval enables tool use, reflection improves multi-agent collaboration, and reasoning benefits all categories. Conclusions: We discuss applications of agentic LLMs and provide an agenda for further research. Important applications are in medical diagnosis, logistics and financial market analysis. Meanwhile, self-reflective agents playing roles and interacting with one another augment the process of scientific research itself. Further, agentic LLMs provide a solution for the problem of LLMs running out of training data: inference-time behavior generates new training states, such that LLMs can keep learning without needing ever larger datasets. We note that there is risk associated with LLM assistants taking action in the real world-safety, liability and security are open problems-while agentic LLMs are also likely to benefit society.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 189 of its 404 references that are in the map. First posted on arXiv 29 Mar 2025.

Builds on

189 of 404 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Practical Considerations for Agentic LLM Systems5 Dec
  2. 2024BenchAgents: Multi-Agent Systems for Structured Benchmark Creation29 Oct
  3. 2024LLM-based multi-agent poetry generation in non-cooperative environments5 Sep
  4. 2024Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Libraries31 Jul
  5. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning9 Jun
  6. 2024ChatShop: Interactive Information Seeking with Language Agents15 Apr
  7. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment19 Feb
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
  9. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agent14 Jan
  10. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents18 Oct
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  12. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
1 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 176

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  13. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  14. 2016OpenAI Gymcited by 1,684
  15. 2018World Modelscited by 1,444
  16. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  17. 2025s1: Simple test-time scalingcited by 1,311
  18. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  19. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  20. 2023Mastering Diverse Domains through World Modelscited by 1,193
  21. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  22. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  23. 2017Mixed Precision Trainingcited by 1,135
  24. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  25. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  26. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  27. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  28. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  29. 2020Mastering Atari with Discrete World Modelscited by 990
  30. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  31. 2023A Survey on Multimodal Large Language Modelscited by 898
  32. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  33. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  34. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  35. 2022A Survey on In-context Learningcited by 717
  36. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  37. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  38. 2024OLMo: Accelerating the Science of Language Modelscited by 599

50 most cited of 176 All 176 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 42; 2026: 63.

105 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 105
  5. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  6. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  7. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  8. 2025Measuring Agents in Productioncited by 35
  9. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  10. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  11. 2025On the Faithfulness of Visual Thinking: Measurement and Enhancementcited by 17
  12. 2026TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Taskscited by 15
38 more
  1. 2025The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systemscited by 14
  2. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  3. 2025ParaView-MCP: An Autonomous Visualization Agent with Direct Tool Usecited by 11
  4. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 11
  5. 2025CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RLcited by 8
  6. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 8
  7. 2025ID-RAG: Identity Retrieval-Augmented Generation for Long-Horizon Persona Coherence in Generative Agentscited by 7
  8. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  9. 2026REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Tracescited by 7
  10. 2025AutoEDA: Enabling EDA Flow Automation through Microservice-Based LLM Agentscited by 6
  11. 2025Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memorycited by 6
  12. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  13. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  14. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  15. 2026Co-Evolving Skill Generation and Policy Optimizationcited by 5
  16. 2026CausalGame: Benchmarking Causal Thinking of LLM Agents in Gamescited by 5
  17. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
  18. 2026The Path Ahead for Agentic AI: Challenges and Opportunitiescited by 4
  19. 2026Agentic Design Patterns: A System-Theoretic Frameworkcited by 4
  20. 2026Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Modelscited by 4
  21. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  22. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?cited by 4
  23. 2025Multi-View Encoders for Performance Prediction in LLM-Based Agentic Workflowscited by 3
  24. 2025KonfAI: A Modular and Fully Configurable Framework for Deep Learning in Medical Imagingcited by 3
  25. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarioscited by 3
  26. 2025Using Copilot Agent Mode to Automate Library Migration: A Quantitative Assessmentcited by 3
  27. 2026Towards Agentic Intelligence for Materials Sciencecited by 3
  28. 2026Agent2Agent Threats in Safety-Critical LLM Assistants: A Human-Centric Taxonomycited by 3
  29. 2026CausalEvolve: Towards Open-Ended Discovery with Causal Scratchpadcited by 3
  30. 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environmentscited by 3
  31. 2026Physics-in-the-Loop: A Hybrid Agentic Architecture for Validated CAD Engineering Designcited by 3
  32. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  33. 2025Baba is LLM: Reasoning in a Game with Dynamic Rulescited by 2
  34. 2025To Trade or Not to Trade: An Agentic Approach to Estimating Market Risk Improves Trading Decisionscited by 2
  35. 2025An Agentic AI for a New Paradigm in Business Process Developmentcited by 2
  36. 2025Neuro-Symbolic Frameworks: Conceptual Characterization and Empirical Comparative Analysiscited by 2
  37. 2025Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrationscited by 2
  38. 2025Refine Thought: A Test-Time Inference Method for Embedding Model Reasoningcited by 2

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  3. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios
  4. 2025Literature Review Of Multi-Agent Debate For Problem-Solving
  5. 2023A Survey on Large Language Model based Autonomous Agents
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges
  7. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects
  8. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Review