mlsanityData to 7 Oct 2026

General LLM agent frameworks and benchmarks

2,917 papers19 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: agents, agent, language, llm, reasoning

Key facts. General LLM agent frameworks and benchmarks: 2,917 papers on arXiv since 2022 (data through 7 Oct 2026). 520 of its 2,917 mapped papers (18%) cite “MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework” (2023, arXiv 2308.00352). 370 of its 2,917 mapped papers (13%) cite “A Survey on Large Language Model based Autonomous Agents” (2023, arXiv 2308.11432). 128 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

128 new papers, 17 Sep – 7 Oct; 19 this week.

  1. This week · 19
  2. Plan-and-Patch: Diffusion Language Models for Agentic Planning also fits: Reinforcement learning for LLM agents
  3. Verification and Self-Improvement in Agentic AI: Foundations and Limits
  4. A Society of Researchers: Designing Institutions for Populations of Autonomous Research Agents also fits: Web, search, and research agents
  5. Agentic AI-Assisted Modeling for Production Scheduling: Assessment in Constraint Programming
  6. AgentTime: Can Agents Estimate and Control Their Own Runtime? also fits: Software engineering coding agents
  7. Not Every Call Needs a Frontier Model: Per-Call-Site Evaluation of Small Language Models in a Deployed Agentic Home-Automation System also fits: Tool learning and function calling
  8. Sequential Probabilistic Uncertainty Estimation for Parallel Multi-Agent Reasoning Systems
  9. Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts? also fits: Tool learning and function calling
  10. Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations also fits: Social simulation and multi-agent debate
  11. Token-Efficient Multi-Agent Collaboration via System One-Guided Computational Division of Labor
All 128
  1. Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents also fits: Agent threats, guardrails and governance
  2. WorkflowOps: Learning Agent Collaboration Priors for Multi-Agent Workflow Orchestration also fits: Software engineering coding agents
  3. DHCG: Dynamic Construction of Hierarchical Collaboration Graphs for LLM-Based Multi-Agent Reasoning
  4. Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents also fits: Agentic AI for wireless networks
  5. ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience? also fits: Reinforcement learning for LLM agents
  6. HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention also fits: Software engineering coding agents
  7. From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation also fits: Reinforcement learning for LLM agents
  8. Attention Tax, Handoff Tax: A Stylised Model of When Multi-Agent LLM Systems Help also fits: Agent memory systems
  9. VERA: Scaling Verifiable Environments for Agentic co-Evolution also fits: Agent skill learning and evolution
  10. Earlier · 109
  11. Safe Context Switching for Agents in the Wild: Mitigating Subspace Interference via Orthogonal Adaptation also fits: Agent memory systems
  12. TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training also fits: Agent skill learning and evolution
  13. Building LLM Agent Systems the Deep Learning Way: From Modular Design to Architecture Search
  14. Assembling Insights for Agentic Machine Learning Engineering Systems also fits: Web, search, and research agents
  15. Agent Behavior as Code: Efficient and Robust LLM Agents with Programmatic Specifications also fits: Software engineering coding agents
  16. Principles that Guide, Actions that Inform: Agent Evolution via Knowledge Abstraction also fits: Reinforcement learning for LLM agents
  17. Organising Trajectory Evidence for Language-Model Agent Assurance: Fragments, Methods, and the Residual also fits: Agent threats, guardrails and governance
  18. MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
  19. Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
  20. Pareto-Dominant Clarification: Post-Training Coding LLMs via PPO-Lagrangian Budget Constraints also fits: Software engineering coding agents
  21. ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models
  22. Dynamic Expert Pruning for Multi-Agent Systems
  23. Harness-Aware Distillation for Small Language Model Agents also fits: Reinforcement learning for LLM agents
  24. DeReAct: Decomposed Reasoning and Acting for Reliable AI Agents also fits: Reinforcement learning for LLM agents
  25. Mimir: Physics-Grounded LLM Agents for Long-Horizon Irrigation Control also fits: Reinforcement learning for LLM agents
  26. Agents Are Systems, Not Models: Rethinking Agentic Evaluation also fits: Software engineering coding agents
  27. Federated Agent Optimization
  28. Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems also fits: Social simulation and multi-agent debate
  29. Cybernetic and Epistemic: A Missing Vocabulary for Trustworthy Agentic Delegation also fits: Agent threats, guardrails and governance
  30. Beyond Leaderboards: Tokenomics of Agentic Small Language Model Ensembles
  31. Finding the Right Fit: Model-Harness Interactions across Agent Tasks also fits: Software engineering coding agents
  32. ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality also fits: Social simulation and multi-agent debate
  33. Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard also fits: Social simulation and multi-agent debate
  34. JevSpawn: Adaptive Agentic Inference through Compositional Action Spaces also fits: Reinforcement learning for LLM agents
  35. Learning from Research: Toward Lifelong Agent Harness Evolution also fits: Agent skill learning and evolution
  36. GraphMAS: A Systematic Benchmark of Multi-Agent Coordination for Graph Learning also fits: Web, search, and research agents
  37. Schema: Discovering Unknown Environments via Agentic Program Induction also fits: Agent skill learning and evolution
  38. Consistent Plan-Act for Long-Horizon Agentic Tasks also fits: Reinforcement learning for LLM agents
  39. Talk2Agent: Benchmarking Voice Interfaces for Text Agents
  40. Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimization also fits: Reinforcement learning for LLM agents
  41. SimpleEvol: An Agent-Loop Framework for LLM-Driven Automated Heuristic Design with Minimal Human Priors
  42. LLM Parkinsonism: Executive-Control Failure, Token-Inefficient Persistence, and an Uncertainty-Aware Global Executive Control Architecture for Autonomous Language-Model Agents
  43. Bayes-Sufficient Compression Is Not Enough: How Does Communication Help Multi-Agent Systems? also fits: Software engineering coding agents
  44. CollabFlow: Recursive Self-Improvement of Agent Collaboration also fits: Agent skill learning and evolution
  45. EvoSteer: Online Self-Evolving Graph Orchestration via Reference-Anchored Credit Assignment also fits: Agent skill learning and evolution
  46. AgBench: Agentic AI Benchmarks for Personal AI Devices also fits: Software engineering coding agents
  47. Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent also fits: Software engineering coding agents
  48. Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agents also fits: Agent memory systems
  49. MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution also fits: Web, search, and research agents
  50. AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks also fits: Reinforcement learning for LLM agents
  51. Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning also fits: Software engineering coding agents
  52. Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution also fits: Software engineering coding agents
  53. Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym also fits: Social simulation and multi-agent debate
  54. AnyAct: Universal Action for Self-Evolving Agents
  55. Harness Evolution as Learning: Approximation, Generalization, and Optimization Limits of Self-Improving Personal Agents also fits: Social simulation and multi-agent debate
  56. Adaptive Consistency Graph for Long-Horizon Agents also fits: Reinforcement learning for LLM agents
  57. Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs also fits: Agent memory systems
  58. When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test also fits: Social simulation and multi-agent debate
  59. Learn Now, Use Next, Trust Later: Prequential Test-Time Learning for LLM Agents also fits: Reinforcement learning for LLM agents
  60. Reinforcing Agentic Creativity in Scientific Ideation with Night Science also fits: Educational agents and math problem solving
  61. LLMs are General Asynchronous Agents also fits: Agent memory systems
  62. Self-Adapting Group of Experts for Multi-Agent Reasoning
  63. AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop? also fits: Agent skill learning and evolution
  64. RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agents
  65. AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs also fits: Software engineering coding agents
  66. LLMs for Executable Multi-Agent System Specification Generation
  67. The Marathon of Scientific Reasoning: Robustness of Scientific Agents to Perturbations in Multi-Turn Interactions also fits: Web, search, and research agents
  68. Org-Agent: Beyond Personal Assistants Towards Organizational Agents
  69. SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents
  70. Waggle: Learning One Anonymous Local Law for Self-Organizing LLM Swarms also fits: Reinforcement learning for LLM agents
  71. Beyond Solo and Consistency: Vindicating Multi-Agent Debate via Conditional Progressive Pruning also fits: Social simulation and multi-agent debate
  72. Prospective Interpretation Risk: Principled Communication Control Between LLMs
  73. DEALS: Decentralized Expertise-Aware Load Serving for Multi-Agent LLM Systems
  74. Raven: The Harness of Harnesses for Composable Agentic Intelligence
  75. On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation also fits: Agent threats, guardrails and governance
  76. Environmental Impact of Generative and Agentic AI: An in-Depth Analysis and Green Solutions also fits: Software engineering coding agents
  77. Theory of Scene: Breaking the Symmetry Trap in Multi-Agent LLM Coordination also fits: Social simulation and multi-agent debate
  78. Planner-as-Router: Joint Plan-Time Model Routing for Cost-Efficient Multi-Agent Workflows also fits: Tool learning and function calling
  79. World Agent: Can Language Models Keep a World Running?
  80. When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents also fits: Agent memory systems
  81. GLIDE: Generalized Layer-wise Intrinsic Distributional Evaluation for Heterogeneous LLM Agents also fits: Reinforcement learning for LLM agents
  82. Clarify the User or Verify the World? Uncertainty Routing for Proactive Agents
  83. HM-ROUTER: Joint Model and Harness Routing for Agentic Systems also fits: Reinforcement learning for LLM agents
  84. Symbolic Guidance for LLM Agents in Distributed Multiagent Coordination
  85. Agents Can Use Base Models to Evade AI Detection also fits: Agent threats, guardrails and governance
  86. Multi-agent Scaling Across Disjunctive and Compensatory Tasks also fits: Social simulation and multi-agent debate
  87. Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes
  88. Task-Oriented Key-Layer KV Communication for Efficient Latent Multi-Agent Collaboration also fits: Agent memory systems
  89. GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI also fits: Reinforcement learning for LLM agents
  90. Who Holds the Pen? Let Specifications, Not Agents, Sign Off also fits: Agent skill learning and evolution
  91. Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents also fits: Reinforcement learning for LLM agents
  92. Breaking the Environment Wall: Evolving LLM Agent Environments for Recursive Self-Improvement also fits: Reinforcement learning for LLM agents
  93. Epistemic-Probabilistic Model for Guarded Multi-Agent LLM Coordination
  94. MeshHeal: Two-Timescale Self-Healing for Gray Failures in Decentralized LLM Agent Networks also fits: Agent skill learning and evolution
  95. Agent-Editing World Model: Rethinking World Modeling for LLM Agents
  96. Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers
  97. Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents also fits: Reinforcement learning for LLM agents
  98. Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms also fits: Reinforcement learning for LLM agents
  99. KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling also fits: Tool learning and function calling
  100. Do We Need Complex Topology Control? Distinct-Peer Random Routing Improves Cost-Efficiency in Sparse Multi-Agent Debate also fits: Social simulation and multi-agent debate
  101. Agensh: Scaling Organizational Intelligence to 1,024 Agents also fits: Software engineering coding agents
  102. REFLEX with Jev for Efficient Selective Control in LLM Agents also fits: Reinforcement learning for LLM agents
  103. The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks also fits: Web, search, and research agents
  104. Recovering Agentic Sovereignty: Mitigating the Consensus Paradox via Contrastive Epistemic Decoding also fits: Social simulation and multi-agent debate
  105. Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction also fits: Multimodal and embodied agents
  106. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction also fits: Reinforcement learning for LLM agents
  107. GRUET: Quantifying Uncertainty of Agentic Reasoning-and-Acting Processes
  108. PhysAI-Bench: A Benchmark for LLM-Based Agentic Decision-Making in Autonomous UAV-Centric Physical AI also fits: Agentic AI for wireless networks
  109. AgentBetta: Verification-Driven Adaptive Configuration of an AI Nano-Agent through Selective Expansion and Verified Contraction
  110. BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents also fits: Tool learning and function calling
  111. Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving also fits: Agent memory systems
  112. OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems
  113. Scaling Discovery through Test-Time Communication also fits: Web, search, and research agents
  114. MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents also fits: Tool learning and function calling
  115. Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics
  116. Rethinking Multi-Agent Collaboration: When More Is Less also fits: Web, search, and research agents
  117. AURORA: A Natural Language-Driven Agentic Framework for Understanding, Reasoning, and Orchestrating Reliable Air-Ground Co-Simulation also fits: Software engineering coding agents
  118. A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems also fits: Social simulation and multi-agent debate
  119. An Architecture for Long-Horizon Agents: Levels, Ticks and Cascaded Intelligence also fits: Reinforcement learning for LLM agents

Foundations

Most cited by papers here

  1. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework2023cited by 520 here
  2. A Survey on Large Language Model based Autonomous Agents2023cited by 370 here
  3. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society2023cited by 365 here
  4. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains2024cited by 264 here
  5. AgentBench: Evaluating LLMs as Agents2023cited by 245 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 435 here
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 192 here
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 175 here
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 305 here
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 520 here
  6. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 409 here
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 370 here
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 243 here
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 198 here
15 more it builds on, not in this chain
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 212 here
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 576 here
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 365 here
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 460 here
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 296 here
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 226 here
  7. 2023ChatDev: Communicative Agents for Software Developmentcited by 223 here
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 312 here
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 245 here
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 162 here
  11. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 130 here
  12. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 206 here
  13. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 190 here
  14. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 173 here
  15. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 264 here

The 24 most cited of 4,434.

Browse

30 of 2,917, ranked 6–35

  1. 6The Rise and Potential of Large Language Model Based Agents: A Survey2023243
  2. 7AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors2023206
  3. 8Large Language Model based Multi-Agents: A Survey of Progress and Challenges2024190
  4. 9ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate2023162
  5. 10ExpeL: LLM Agents Are Experiential Learners2023130
  6. 11A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration2023116
  7. 12Mixture-of-Agents Enhances Large Language Model Capabilities2024107
  8. 13Multi-Agent Collaboration Mechanisms: A Survey of LLMs2025106
  9. 14TravelPlanner: A Benchmark for Real-World Planning with Language Agents2024105
  10. 15Scaling Large Language Model-based Multi-Agent Collaboration2024102
All 30
  1. 16Automated Design of Agentic Systems202489
  2. 17Understanding the planning of LLM agents: A survey202488
  3. 18Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models202384
  4. 19AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents202482
  5. 20Cognitive Architectures for Language Agents202382
  6. 21AgentTuning: Enabling Generalized Agent Abilities for LLMs202377
  7. 22Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration202374
  8. 23Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents202372
  9. 24TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks202471
  10. 25Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems202468
  11. 26AutoAgents: A Framework for Automatic Agent Generation202368
  12. 27FireAct: Toward Language Agent Fine-tuning202364
  13. 28Building Cooperative Embodied Agents Modularly with Large Language Models202363
  14. 29SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks202361
  15. 30Multi-agent Architecture Search via Agentic Supernet202560
  16. 31MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents202560
  17. 32Chain of Agents: Large Language Models Collaborating on Long-Context Tasks202458
  18. 33G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks202458
  19. 34Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents202353
  20. 35Language Agents as Optimizable Graphs202452