mlsanityData to 14 Oct 2026

Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

arXiv 2606.20014 ↗First posted 18 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning (RL) has achieved strong performance in sequential decision-making, yet scaling to complex multi-agent environments remains challenging due to sparse rewards, large state-action spaces, and the difficulty of learning coordinated strategies. We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a centralized strategic controller that selects among specialized RL skill policies for a team of agents, while RL policies handle reactive low-level execution. We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and ``Flat'' RL (end-to-end training without skill decomposition) baselines. The LLM+RL system achieves task performance statistically equivalent to hand-crafted BT (46.4% vs 51.5% win rate, p=0.103) while both significantly outperform Flat RL trained without skill decomposition. A user study (n=15) reveals that 60% of participants perceive LLM+RL agents as the most human-like (p=0.027), citing behavioral adaptability and tactical variability. These results demonstrate that pretrained LLM reasoning can effectively orchestrate pretrained RL skills, achieving competitive multi-agent coordination and superior perceived believability without manual rule engineering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 19 of its 54 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

19 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Self-correcting Reward Shaping via Language Models for Reinforcement Learning Agents in Games30 Jun
  2. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policy13 Mar

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2023A Comprehensive Overview of Large Language Modelscited by 648
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  10. 2016The Option-Critic Architecturecited by 448
  11. 2024Reinforcement learningcited by 247
  12. 2018Unity: A General Platform for Intelligent Agentscited by 236
5 more
  1. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  2. 2017Behavior Trees in Robotics and AI: An Introductioncited by 107
  3. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learningcited by 22
  4. 2024Reinforcement Learning for High-Level Strategic Control in Tower Defense Gamescited by 3
  5. 2026Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
  2. 2026Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments
  3. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents
  4. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
  5. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  6. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  7. 2024Large Language Models as Agents in Two-Player Games
  8. 2026Hierarchical Lead Critic based Multi-Agent Reinforcement Learning