# Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

arXiv: https://arxiv.org/abs/1609.02993 · HTML: https://mlsanity.com/paper/1609.02993/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2016.

## Built on by (most cited first, 7 of 7)

- [On Efficient Reinforcement Learning for Full-length Game of StarCraft II](https://mlsanity.com/paper/2209.11553/) (2022)
- [An Introduction of mini-AlphaStar](https://mlsanity.com/paper/2104.06890/) (2021)
- [Concentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems](https://mlsanity.com/paper/2203.06416/) (2022)
- [A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learning](https://mlsanity.com/paper/2208.03002/) (2022)
- [A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2208.07753/) (2022)
- [Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graph](https://mlsanity.com/paper/2403.18056/) (2024)
- [Mix Q-learning for Lane Changing: A Collaborative Decision-Making Method in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2406.09755/) (2024)

## Abstract (arXiv)

We consider scenarios from the real-time strategy game StarCraft as new benchmarks for reinforcement learning algorithms. We propose micromanagement tasks, which present the problem of the short-term, low-level control of army members during a battle. From a reinforcement learning point of view, these scenarios are challenging because the state-action space is very large, and because there is no obvious feature representation for the state-action evaluation function. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. In addition, we present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm allows for the collection of traces for learning using deterministic policies, which appears much more efficient than, for example, ε-greedy exploration. Experiments show that with this algorithm, we successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.