# On Efficient Reinforcement Learning for Full-length Game of StarCraft II

arXiv: https://arxiv.org/abs/2209.11553 · HTML: https://mlsanity.com/paper/2209.11553/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II](https://mlsanity.com/paper/2012.13169/) (2020)
- [TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game](https://mlsanity.com/paper/2011.13729/) (2020)
- [ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games](https://mlsanity.com/paper/1707.01067/) (2017)
- [On Reinforcement Learning for Full-length Game of StarCraft](https://mlsanity.com/paper/1809.09095/) (2018)
- [Accuracy-based Curriculum Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/1806.09614/) (2018)
- [TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game](https://mlsanity.com/paper/1809.07193/) (2018)
- [Efficient Reinforcement Learning for StarCraft by Abstract Forward Models and Transfer Learning](https://mlsanity.com/paper/1903.00715/) (2019)
- [An Introduction of mini-AlphaStar](https://mlsanity.com/paper/2104.06890/) (2021)
- [Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks](https://mlsanity.com/paper/1609.02993/) (2016)
- [Learning Macromanagement in StarCraft from Replays using Deep Learning](https://mlsanity.com/paper/1707.03743/) (2017)
- [Rethinking of AlphaStar](https://mlsanity.com/paper/2108.03452/) (2021)

## Built on by (most cited first, 14 of 14)

- [SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models](https://mlsanity.com/paper/2401.17749/) (2024)
- [SMAC-R1: The Emergence of Intelligence in Decision-Making Tasks](https://mlsanity.com/paper/2410.16024/) (2024)
- [Reflection of Episodes: Learning to Play Game from Expert and Self Experiences](https://mlsanity.com/paper/2502.13388/) (2025)
- [Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis](https://mlsanity.com/paper/2505.12462/) (2025)
- [EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-making](https://mlsanity.com/paper/2508.09586/) (2025)
- [Adaptive Command: Real-Time Policy Adjustment via Language Models in StarCraft II](https://mlsanity.com/paper/2508.16580/) (2025)
- [Optimizing Quantum Circuits via ZX Diagrams using Reinforcement Learning and Graph Neural Networks](https://mlsanity.com/paper/2504.03429/) (2025)
- [TacticCraft: Natural Language-Driven Tactical Adaptation for StarCraft II](https://mlsanity.com/paper/2507.15618/) (2025)
- [Element-Wise Attention Layers: an option for optimization](https://mlsanity.com/paper/2302.05488/) (2023)
- [Recent Trends in Artificial Intelligence Technology: A Scoping Review](https://mlsanity.com/paper/2305.04532/) (2023)
- [StarBench: A Turn-Based RPG Benchmark for Agentic Multimodal Decision-Making and Information Seeking](https://mlsanity.com/paper/2510.18483/) (2025)
- [Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay](https://mlsanity.com/paper/2512.05320/) (2025)
- [Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmark](https://mlsanity.com/paper/2603.06608/) (2026)
- [Hierarchical Reinforcement Learning in StarCraft Micromanagement with Influence Maps and Cluster-based Scripts](https://mlsanity.com/paper/2606.30092/) (2026)

## Abstract (arXiv)

StarCraft II (SC2) poses a grand challenge for reinforcement learning (RL), of which the main difficulties include huge state space, varying action space, and a long time horizon. In this work, we investigate a set of RL techniques for the full-length game of StarCraft II. We investigate a hierarchical RL approach involving extracted macro-actions and a hierarchical architecture of neural networks. We investigate a curriculum transfer training procedure and train the agent on a single machine with 4 GPUs and 48 CPU threads. On a 64x64 map and using restrictive units, we achieve a win rate of 99% against the level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat models, we achieve a 93% win rate against the most difficult non-cheating level built-in AI (level-7). In this extended version of the paper, we improve our architecture to train the agent against the cheating level AIs and achieve the win rate against the level-8, level-9, and level-10 AIs as 96%, 97%, and 94%, respectively. Our codes are at https://github.com/liuruoze/HierNet-SC2. To provide a baseline referring the AlphaStar for our work as well as the research and open-source community, we reproduce a scaled-down version of it, mini-AlphaStar (mAS). The latest version of mAS is 1.07, which can be trained on the raw action space which has 564 actions. It is designed to run training on a single common machine, by making the hyper-parameters adjustable. We then compare our work with mAS using the same resources and show that our method is more effective. The codes of mini-AlphaStar are at https://github.com/liuruoze/mini-AlphaStar. We hope our study could shed some light on the future research of efficient reinforcement learning on SC2 and other large-scale games.