mlsanityData to 14 Oct 2026

Hierarchical Reinforcement Learning in StarCraft Micromanagement with Influence Maps and Cluster-based Scripts

arXiv 2606.30092 ↗First posted 29 Jun 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Real-time strategy (RTS) games present significant AI challenges, characterized by expansive state-action spaces arising from multi-unit coordination in continuous battlefields, and sparse delayed rewards stemming from final win/lose signals. Existing approaches face a trade-off between managing the dimensionality explosion of joint actions and maintaining the interpretability of complex state representations. This complexity is further intensified by the limitation of traditional hierarchical structures in adaptively decomposing tasks into effective tactical modules. Such difficulties are compounded by the black-box nature of deep learning models and their reliance on sparse rewards, which together result in limited sample efficiency and a lack of decision-making transparency. To address these limitations, this paper proposes HRL-IM/CBS, a hierarchical reinforcement learning framework with influence map hashing and cluster-based scripts for StarCraft micromanagement. Influence map hashing encodes global battlefield situations into compact hexadecimal codes, capturing spatial control and relative advantage. Cluster-based scripts enable dynamic local coordination through adaptive unit partitioning. The hierarchical multi-Q-table architecture decomposes decision-making into upper-level clustering strategy selection and lower-level tactical execution, with reward allocation providing dense learning signals. Experiments across six asymmetric scenarios demonstrate competitive performance against deep RL baselines while offering advantages in sample efficiency and interpretability through transparent Q-table representations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 55 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

16 of 55 references.

References · 16

  1. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  2. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  3. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  4. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  5. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  6. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
  7. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 96
  8. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  9. 2018StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learningcited by 25
  10. 2020TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Gamecited by 21
  11. 2022On Efficient Reinforcement Learning for Full-length Game of StarCraft IIcited by 14
  12. 2018Macro action selection with deep reinforcement learning in StarCraftcited by 6
4 more
  1. 2017Asymmetric Action Abstractions for Multi-Unit Control in Adversarial Real-Time Gamescited by 3
  2. 2023CODEX: A Cluster-Based Method for Explainable Reinforcement Learningcited by 3
  3. 2022Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperationcited by 1
  4. 2025Three-dimensional attention Transformer for state evaluation in real-time strategy gamescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making
  2. 2019Hierarchical Reinforcement Learning for Multi-agent MOBA Game
  3. 2025A Hierarchical Hybrid AI Approach: Integrating Deep Reinforcement Learning and Scripted Agents in Combat Simulations
  4. 2024Mastering the Digital Art of War: Developing Intelligent Combat Simulation Agents for Wargaming Using Hierarchical Reinforcement Learning
  5. 2024MAIDCRL: Semi-centralized Multi-Agent Influence Dense-CNN Reinforcement Learning
  6. 2020SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
  7. 2026World Models for Policy Refinement in StarCraft II
  8. 2025Society of Mind Meets Real-Time Strategy: A Hierarchical Multi-Agent Framework for Strategic Reasoning