# On Reinforcement Learning for Full-length Game of StarCraft

arXiv: https://arxiv.org/abs/1809.09095 · HTML: https://mlsanity.com/paper/1809.09095/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2018.

## Built on by (most cited first, 16 of 16)

- [MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions](https://mlsanity.com/paper/2410.02743/) (2024)
- [On Efficient Reinforcement Learning for Full-length Game of StarCraft II](https://mlsanity.com/paper/2209.11553/) (2022)
- [An Introduction of mini-AlphaStar](https://mlsanity.com/paper/2104.06890/) (2021)
- [Deep Learning vs. Gradient Boosting: Benchmarking state-of-the-art machine learning algorithms for credit scoring](https://mlsanity.com/paper/2205.10535/) (2022)
- [Controlling Large Language Model with Latent Actions](https://mlsanity.com/paper/2503.21383/) (2025)
- [Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03429/) (2023)
- [Reinforcement Learning for High-Level Strategic Control in Tower Defense Games](https://mlsanity.com/paper/2406.07980/) (2024)
- [Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm](https://mlsanity.com/paper/2102.00168/) (2021)
- [Rethinking of AlphaStar](https://mlsanity.com/paper/2108.03452/) (2021)
- [Method for making multi-attribute decisions in wargames by combining intuitionistic fuzzy numbers with reinforcement learning](https://mlsanity.com/paper/2109.02354/) (2021)
- [MIMIC-Py: An Extensible Tool for Personality-Driven Automated Game Testing with Large Language Models](https://mlsanity.com/paper/2604.07752/) (2026)
- [Applying supervised and reinforcement learning methods to create neural-network-based agents for playing StarCraft II](https://mlsanity.com/paper/2109.12691/) (2021)
- [Towards autonomous artificial agents with an active self: modeling sense of control in situated action](https://mlsanity.com/paper/2112.05577/) (2021)
- [Adaptive Control of Resource Flow to Optimize Construction Work and Cash Flow via Online Deep Reinforcement Learning](https://mlsanity.com/paper/2307.10574/) (2023)
- [Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmark](https://mlsanity.com/paper/2603.06608/) (2026)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

StarCraft II poses a grand challenge for reinforcement learning. The main difficulties of it include huge state and action space and a long-time horizon. In this paper, we investigate a hierarchical reinforcement learning approach for StarCraft II. The hierarchy involves two levels of abstraction. One is the macro-action automatically extracted from expert's trajectories, which reduces the action space in an order of magnitude yet remains effective. The other is a two-layer hierarchical architecture which is modular and easy to scale, enabling a curriculum transferring from simpler tasks to more complex tasks. The reinforcement training algorithm for this architecture is also investigated. On a 64x64 map and using restrictive units, we achieve a winning rate of more than 99% against the difficulty level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat model, we can achieve over 93% winning rate of Protoss against the most difficult non-cheating built-in AI (level-7) of Terran, training within two days using a single machine with only 48 CPU cores and 8 K40 GPUs. It also shows strong generalization performance, when tested against never seen opponents including cheating levels built-in AI and all levels of Zerg and Protoss built-in AI. We hope this study could shed some light on the future research of large-scale reinforcement learning.