mlsanityData to 14 Oct 2026

Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learning

arXiv 2405.16854 ↗First posted 27 May 2024cs.MAcited by 3

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Multi-agent reinforcement learning (MARL) is employed to develop autonomous agents that can learn to adopt cooperative or competitive strategies within complex environments. However, the linear increase in the number of agents leads to a combinatorial explosion of the action space, which may result in algorithmic instability, difficulty in convergence, or entrapment in local optima. While researchers have designed a variety of effective algorithms to compress the action space, these methods also introduce new challenges, such as the need for manually designed prior knowledge or reliance on the structure of the problem, which diminishes the applicability of these techniques. In this paper, we introduce Evolutionary action SPAce Reduction with Knowledge (eSpark), an exploration function generation framework driven by large language models (LLMs) to boost exploration and prune unnecessary actions in MARL. Using just a basic prompt that outlines the overall task and setting, eSpark is capable of generating exploration functions in a zero-shot manner, identifying and pruning redundant or irrelevant state-action pairs, and then achieving autonomous improvement from policy feedback. In reinforcement learning tasks involving inventory management and traffic light control encompassing a total of 15 scenarios, eSpark consistently outperforms the combined MARL algorithm in all scenarios, achieving an average performance gain of 34.4% and 9.9% in the two types of tasks respectively. Additionally, eSpark has proven to be capable of managing situations with a large number of agents, securing a 29.7% improvement in scalability challenges that featured over 500 agents. The code can be found in https://github.com/LiuZhihao2022/eSpark.git.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 58 of its 81 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

58 of 81 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory25 May
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 56

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
38 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  3. 2023StarCoder: may the source be with you!cited by 799
  4. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  6. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  7. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  8. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  9. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  10. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  11. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
  12. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 276
  13. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  14. 2023Reward Design with Language Modelscited by 261
  15. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  16. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  17. 2019CoLight: Learning Network-level Cooperation for Traffic Signal Controlcited by 138
  18. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 137
  19. 2021Skill Induction and Planning with Latent Languagecited by 99
  20. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
  21. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  22. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  23. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  24. 2018Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learningcited by 53
  25. 2023Learning to Generate Better Than Your LLMcited by 49
  26. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thoughtcited by 39
  27. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  28. 2023LLMs for Science: Usage for Code Generation and Data Analysiscited by 34
  29. 2017Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learningcited by 33
  30. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  31. 2019ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learningcited by 17
  32. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  33. 2021Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Explorationcited by 14
  34. 2017What can you do with a rock? Affordance extraction via word embeddingscited by 13
  35. 2023From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARLcited by 11
  36. 2023A Versatile Multi-Agent Reinforcement Learning Benchmark for Inventory Managementcited by 10
  37. 2020Multi-Agent Meta-Reinforcement Learning for Self-Powered and Sustainable Edge Computing Systemscited by 9
  38. 2023GPT-in-the-Loop: Adaptive Decision-Making for Multiagent Systemscited by 9

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 34
  2. 2025LLM-Based Social Simulations Require a Boundarycited by 12
  3. 2024MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiationcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning
  2. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  3. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Exploration
  4. 2026LangMARL: Natural Language Multi-Agent Reinforcement Learning
  5. 2024Empowering Large Language Model Agents through Action Learning
  6. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  7. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  8. 2024Reinforcing Language Agents via Policy Optimization with Action Decomposition