mlsanityData to 7 Oct 2026

From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review

arXiv 2504.19678 ↗First posted 28 Apr 2025cs.AI cs.LGcited by 146

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large language models and autonomous AI agents have evolved rapidly, resulting in a diverse array of evaluation benchmarks, frameworks, and collaboration protocols. Driven by the growing need for standardized evaluation and integration, we systematically consolidate these fragmented efforts into a unified framework. However, the landscape remains fragmented and lacks a unified taxonomy or comprehensive survey. Therefore, we present a side-by-side comparison of benchmarks developed between 2019 and 2025 that evaluate these models and agents across multiple domains. In addition, we propose a taxonomy of approximately 60 benchmarks that cover general and academic knowledge reasoning, mathematical problem-solving, code generation and software engineering, factual grounding and retrieval, domain-specific evaluations, multimodal and embodied tasks, task orchestration, and interactive assessments. Furthermore, we review AI-agent frameworks introduced between 2023 and 2025 that integrate large language models with modular toolkits to enable autonomous decision-making and multi-step reasoning. Moreover, we present real-world applications of autonomous AI agents in materials science, biomedical research, academic ideation, software engineering, synthetic data generation, chemical reasoning, mathematical problem-solving, geographic information systems, multimedia, healthcare, and finance. We then survey key agent-to-agent collaboration protocols, namely the Agent Communication Protocol (ACP), the Model Context Protocol (MCP), and the Agent-to-Agent Protocol (A2A). Finally, we discuss recommendations for future research, focusing on advanced reasoning strategies, failure modes in multi-agent LLM systems, automated scientific discovery, dynamic tool integration via reinforcement learning, integrated search capabilities, and security vulnerabilities in agent protocols.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 189 of its 234 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

189 of 234 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents9 Apr
  2. 2025GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analytics27 Mar
  3. 2025Verbal Process Supervision Elicits Better Coding Agents24 Mar
  4. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?16 Mar
  5. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code10 Mar
  6. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents3 Mar
  7. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning16 Feb
  8. 2024Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning29 Oct
  9. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation1 Aug
  10. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving26 Apr
  11. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct
  12. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation2 Oct

Other references · 177

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
38 more
  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  7. 2025Qwen2.5-Omni Technical Reportcited by 704
  8. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  9. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  10. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  11. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  13. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  14. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  15. 2024Understanding the planning of LLM agents: A surveycited by 414
  16. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  17. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  18. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  19. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  20. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  21. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  22. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  23. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  24. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  25. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  26. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  27. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  28. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  29. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 257
  30. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
  31. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  32. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  33. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  34. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  35. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  36. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  37. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  38. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200

50 most cited of 177 All 177 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 51; 2026: 95.

146 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  3. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61
  4. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  5. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  6. 2026EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoningcited by 52
  7. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  8. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  9. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 23
  10. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  11. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 18
  12. 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profilescited by 16
38 more
  1. 2025Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerancecited by 16
  2. 2025WebMall -- A Multi-Shop Benchmark for Evaluating Web Agentscited by 15
  3. 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 15
  4. 2026CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Eracited by 14
  5. 2025Real-Time Reasoning Agents in Evolving Environmentscited by 13
  6. 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agentscited by 13
  7. 2025Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Factscited by 12
  8. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  9. 2026α3-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networkscited by 10
  10. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  11. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  12. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  13. 2025MR.Rec: Synergizing Memory and Reasoning for Personalized Recommendation Assistant with LLMscited by 9
  14. 2025Agentic AI for Intent-Based Industrial Automationcited by 7
  15. 2025From Multi-Agent Systems and the Semantic Web to Agentic AI: A Unified Narrative of the Web of Agentscited by 7
  16. 2025LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Servicescited by 7
  17. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  18. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 7
  19. 2025Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedbackcited by 6
  20. 2025An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editingcited by 6
  21. 2025Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrievalcited by 6
  22. 2025SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMscited by 6
  23. 2026Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agentscited by 6
  24. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learningcited by 6
  25. 2026Brain-Inspired Graph Multi-Agent Systems for LLM Reasoningcited by 6
  26. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5
  27. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  28. 2025AgentSense: LLMs Empower Generalizable and Explainable Web-Based Participatory Urban Sensingcited by 5
  29. 2025When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Modelscited by 5
  30. 2025Toward Autonomous Engineering Design: A Knowledge-Guided Multi-Agent Frameworkcited by 5
  31. 2026Large Language Model Agent for User-friendly Chemical Process Simulationscited by 5
  32. 2026Exploiting LLM Agent Supply Chains via Payload-less Skillscited by 5
  33. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 4
  34. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  35. 2025Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentationcited by 4
  36. 2025Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metricscited by 4
  37. 2025A Tutorial on Cognitive Biases in Agentic AI-Driven 6G Autonomous Networkscited by 4
  38. 2025MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)cited by 4

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Large Language Model based Autonomous Agents
  2. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios
  3. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  4. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Survey
  6. 2026The Path Ahead for Agentic AI: Challenges and Opportunities
  7. 2025The FM Agent
  8. 2025Evaluation and Benchmarking of LLM Agents: A Survey