mlsanityData to 14 Oct 2026

LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions

arXiv 2405.11106 ↗First posted 17 May 2024cs.MA cs.AI cs.CLcited by 67

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

In recent years, Large Language Models (LLMs) have shown great abilities in various tasks, including question answering, arithmetic problem solving, and poem writing, among others. Although research on LLM-as-an-agent has shown that LLM can be applied to Reinforcement Learning (RL) and achieve decent results, the extension of LLM-based RL to Multi-Agent System (MAS) is not trivial, as many aspects, such as coordination and communication between agents, are not considered in the RL frameworks of a single agent. To inspire more research on LLM-based MARL, in this letter, we survey the existing LLM-based single-agent and multi-agent RL frameworks and provide potential research directions for future research. In particular, we focus on the cooperative tasks of multiple agents with a common goal and communication among them. We also consider human-in/on-the-loop scenarios enabled by the language component in the framework.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 65 of its 85 references that are in the map. First posted on arXiv 17 May 2024.

Builds on

65 of 85 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments18 Mar
  2. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization4 Aug
  3. 2023REX: Rapid Exploration and eXploitation for AI Agents18 Jul
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  6. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  7. 2020Aligning AI With Shared Human Valuescited by 723
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  9. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  10. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  11. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  12. 2016Learning Multiagent Communication with Backpropagationcited by 366
  13. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  14. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  15. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  16. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
  17. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  18. 2018TarMAC: Targeted Multi-Agent Communicationcited by 208
  19. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  20. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  21. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 177
  22. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  23. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 175
  24. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  25. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 170
  26. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  27. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  28. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  29. 2022Learning by Distilling Contextcited by 138
  30. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 135
  31. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  32. 2018Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applicationscited by 127
  33. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  34. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 98
  35. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 82
  36. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  37. 2017VAIN: Attentional Multi-agent Predictive Modelingcited by 53
  38. 2023Multi-Agent Consensus Seeking via Large Language Modelscited by 50

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 37; 2026: 18.

67 papers.

Sort:

By area:

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  5. 2024Emergent social conventions and collective bias in LLM populationscited by 123
  6. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 90
  7. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  8. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 44
  9. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  10. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  11. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32
  12. 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Modelscited by 32
38 more
  1. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  2. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  3. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  4. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  5. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 18
  6. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 15
  7. 2024Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learningcited by 14
  8. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 14
  9. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  10. 2025Learning from Diverse Reasoning Paths with Routing and Collaborationcited by 12
  11. 2025Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Surveycited by 10
  12. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  13. 2024YOLO-MARL: You Only LLM Once for Multi-Agent Reinforcement Learningcited by 9
  14. 2025RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integrationcited by 9
  15. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  16. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9
  17. 2025DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discoverycited by 9
  18. 2024Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMscited by 8
  19. 2025RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarmscited by 7
  20. 2025STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planningcited by 6
  21. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  22. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoningcited by 6
  23. 2024MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiationcited by 5
  24. 2025Empowering Medical Multi-Agents with Clinical Consultation Flow for Dynamic Diagnosiscited by 5
  25. 2025MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computingcited by 5
  26. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  27. 2026Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awarenesscited by 5
  28. 2025Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignmentcited by 4
  29. 2025LLM-Mediated Guidance of MARL Systemscited by 3
  30. 2025OMAC: A Holistic Optimization Framework for LLM-Based Multi-Agent Collaborationcited by 3
  31. 2025From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Contextcited by 3
  32. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordinationcited by 3
  33. 2026MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generationcited by 3
  34. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  35. 2025Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agentscited by 2
  36. 2025MasHost Builds It All: Autonomous Multi-Agent System Directed by Reinforcement Learningcited by 2
  37. 2025Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Surveycited by 2
  38. 2025AURA: Adaptive Unified Reasoning and Automation with LLM-Guided MARL for NextG Cellular Networkscited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application
  2. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  3. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects
  4. 2026Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
  5. 2025A Survey on the Optimization of Large Language Model-based Agents
  6. 2023A Survey on Large Language Model based Autonomous Agents
  7. 2025Language-Driven Coordination and Learning in Multi-Agent Simulation Environments
  8. 2024Reinforcement Learning Problem Solving with Large Language Models