mlsanityData to 14 Oct 2026

AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning

arXiv 2308.03526 ↗First posted 7 Aug 2023cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

StarCraft II is one of the most challenging simulated reinforcement learning environments; it is partially observable, stochastic, multi-agent, and mastering StarCraft II requires strategic planning over long time horizons with real-time low-level execution. It also has an active professional competitive scene. StarCraft II is uniquely suited for advancing offline RL algorithms, both because of its challenging nature and because Blizzard has released a massive dataset of millions of StarCraft II games played by human players. This paper leverages that and establishes a benchmark, called AlphaStar Unplugged, introducing unprecedented challenges for offline reinforcement learning. We define a dataset (a subset of Blizzard's release), tools standardizing an API for machine learning methods, and an evaluation protocol. We also present baseline agents, including behavior cloning, offline variants of actor-critic and MuZero. We improve the state of the art of agents using only offline data, and we achieve 90% win rate against previously published AlphaStar behavior cloning agent.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 55 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

36 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022An Empirical Study of Implicit Regularization in Deep Offline RL5 Jul
  2. 2022When does return-conditioned supervised learning work for offline reinforcement learning?2 Jun
  3. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  4. 2021Offline RL Without Off-Policy Evaluation16 Jun
  5. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  6. 2021Regularized Behavior Value Estimation17 Mar
  7. 2020Critic Regularized Regression26 Jun
  8. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
12 more
  1. 2018Learning Dexterous In-Hand Manipulationcited by 687
  2. 2019The StarCraft Multi-Agent Challengecited by 597
  3. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  4. 2019Challenges of Real-World Reinforcement Learningcited by 291
  5. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  7. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  8. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  9. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  10. 2021Learning and Planning in Complex Action Spacescited by 77
  11. 2020TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Gamecited by 21
  12. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 3.

9 papers.

  1. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  2. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  3. 2026From Gameplay Traces to Game Mechanics: Causal Induction with Large Language Modelscited by 2
  4. 2024Offline Fictitious Self-Play for Competitive Gamescited by 1
  5. 2025A Data-Driven Discretized CS:GO Simulation Environment to Facilitate Strategic Multi-Agent Planning Researchcited by 1
  6. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
  7. 2024Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spacescited by 0
  8. 2026Play Like Champions: Counterfactual Feedback Generation in Latent Spacecited by 0
  9. 2026Esports World Model Datasetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II
  2. 2025AVA: Attentive VLM Agent for Mastering StarCraft II
  3. 2026Runtime Action Interference for AI Control of AlphaStar in StarCraft II
  4. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks
  5. 2026World Models for Policy Refinement in StarCraft II
  6. 2025HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  8. 2026Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmark