# Learning Diverse Policies in MOBA Games via Macro-Goals

arXiv: https://arxiv.org/abs/2110.14221 · HTML: https://mlsanity.com/paper/2110.14221/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Deep Counterfactual Regret Minimization](https://mlsanity.com/paper/1811.00164/) (2018)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings](https://mlsanity.com/paper/2011.12582/) (2020)
- [MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks](https://mlsanity.com/paper/2105.06350/) (2021)
- [Hierarchical Macro Strategy Model for MOBA Game AI](https://mlsanity.com/paper/1812.07887/) (2018)
- [Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search](https://mlsanity.com/paper/2012.10171/) (2020)
- [Boosting Offline Reinforcement Learning with Residual Generative Modeling](https://mlsanity.com/paper/2106.10411/) (2021)
- [MOBA: a New Arena for Game AI](https://mlsanity.com/paper/1705.10443/) (2017)

## Built on by (most cited first, 7 of 7)

- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks](https://mlsanity.com/paper/2408.10556/) (2024)
- [Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspective](https://mlsanity.com/paper/2304.11632/) (2023)
- [Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain](https://mlsanity.com/paper/2401.16444/) (2024)
- [RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation](https://mlsanity.com/paper/2505.03344/) (2025)
- [CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies](https://mlsanity.com/paper/2605.04470/) (2026)
- [Offline Imitation of Badminton Player Behavior via Experiential Contexts and Brownian Motion](https://mlsanity.com/paper/2403.12406/) (2024)

## Abstract (arXiv)

Recently, many researchers have made successful progress in building the AI systems for MOBA-game-playing with deep reinforcement learning, such as on Dota 2 and Honor of Kings. Even though these AI systems have achieved or even exceeded human-level performance, they still suffer from the lack of policy diversity. In this paper, we propose a novel Macro-Goals Guided framework, called MGG, to learn diverse policies in MOBA games. MGG abstracts strategies as macro-goals from human demonstrations and trains a Meta-Controller to predict these macro-goals. To enhance policy diversity, MGG samples macro-goals from the Meta-Controller prediction and guides the training process towards these goals. Experimental results on the typical MOBA game Honor of Kings demonstrate that MGG can execute diverse policies in different matches and lineups, and also outperform the state-of-the-art methods over 102 heroes.