mlsanityData to 7 Oct 2026

Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Exploration

arXiv 2410.02511 ↗First posted 3 Oct 2024cs.AI cs.MAcited by 9

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

With expansive state-action spaces, efficient multi-agent exploration remains a longstanding challenge in reinforcement learning. Although pursuing novelty, diversity, or uncertainty attracts increasing attention, redundant efforts brought by exploration without proper guidance choices poses a practical issue for the community. This paper introduces a systematic approach, termed LEMAE, choosing to channel informative task-relevant guidance from a knowledgeable Large Language Model (LLM) for Efficient Multi-Agent Exploration. Specifically, we ground linguistic knowledge from LLM into symbolic key states, that are critical for task fulfillment, in a discriminative manner at low LLM inference costs. To unleash the power of key states, we design Subspace-based Hindsight Intrinsic Reward (SHIR) to guide agents toward key states by increasing reward density. Additionally, we build the Key State Memory Tree (KSMT) to track transitions between key states in a specific task for organized exploration. Benefiting from diminishing redundant explorations, LEMAE outperforms existing SOTA approaches on the challenging benchmarks (e.g., SMAC and MPE) by a large margin, achieving a 10x acceleration in certain scenarios.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 49 of its 78 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

49 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents22 Nov
  2. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents14 Oct
  3. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory25 May
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2019The StarCraft Multi-Agent Challengecited by 589
32 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  2. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  3. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  4. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  5. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  6. 2023Reward Design with Language Modelscited by 257
  7. 2020First return, then explorecited by 256
  8. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  10. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  11. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  12. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  13. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  14. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  15. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  16. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  17. 2019Influence-Based Multi-Agent Explorationcited by 75
  18. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  19. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  20. 2022Exploration via Elliptical Episodic Bonusescited by 51
  21. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  22. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  23. 2023A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environmentcited by 37
  24. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  25. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  26. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  27. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  28. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  29. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  30. 2024RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Modelscited by 11
  31. 2024Efficient Multi-agent Reinforcement Learning by Planningcited by 9
  32. 2024Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 2.

9 papers.

  1. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  2. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 22
  3. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  4. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  5. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 11
  6. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  7. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  8. 2025Curriculum-Based Multi-Tier Semantic Exploration via Deep Reinforcement Learningcited by 0
  9. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  2. 2025Toward Efficient Exploration by Large Language Model Agents
  3. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  4. 2026Look Before You Leap: Autonomous Exploration for LLM Agents
  5. 2026Multi-Agent LLMs Fail to Explore Each Other
  6. 2025Disentangling Exploration of Large Language Models by Optimal Exploitation
  7. 2026Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimization
  8. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models