# Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.08483 · HTML: https://mlsanity.com/paper/2209.08483/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 18 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Distilling Policy Distillation](https://mlsanity.com/paper/1902.02186/) (2019)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings](https://mlsanity.com/paper/2011.12582/) (2020)
- [Hierarchical Macro Strategy Model for MOBA Game AI](https://mlsanity.com/paper/1812.07887/) (2018)
- [Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search](https://mlsanity.com/paper/2012.10171/) (2020)
- [Boosting Offline Reinforcement Learning with Residual Generative Modeling](https://mlsanity.com/paper/2106.10411/) (2021)
- [Feedback-Based Tree Search for Reinforcement Learning](https://mlsanity.com/paper/1805.05935/) (2018)
- [What Makes a Good Team? A Large-scale Study on the Effect of Team Composition in Honor of Kings](https://mlsanity.com/paper/1902.06432/) (2019)
- [Unsupervised Multi-stream Highlight detection for the Game "Honor of Kings"](https://mlsanity.com/paper/1910.06189/) (2019)

## Built on by (most cited first, 28 of 28)

- [Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning](https://mlsanity.com/paper/2308.14284/) (2023)
- [RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization](https://mlsanity.com/paper/2307.10224/) (2023)
- [A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2403.10249/) (2024)
- [Games for Artificial Intelligence Research: A Review and Perspectives](https://mlsanity.com/paper/2304.13269/) (2023)
- [Revisiting Discrete Soft Actor-Critic](https://mlsanity.com/paper/2209.10081/) (2022)
- [Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2306.03034/) (2023)
- [Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Games](https://mlsanity.com/paper/2307.11105/) (2023)
- [Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks](https://mlsanity.com/paper/2408.10556/) (2024)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [Metacognitive AI: Framework and the Case for a Neurosymbolic Approach](https://mlsanity.com/paper/2406.12147/) (2024)
- [Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain](https://mlsanity.com/paper/2401.16444/) (2024)
- [SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMAC](https://mlsanity.com/paper/2412.17707/) (2024)
- [Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments](https://mlsanity.com/paper/2506.06981/) (2025)
- [Probabilistic Offline Policy Ranking with Approximate Bayesian Computation](https://mlsanity.com/paper/2312.11551/) (2023)
- [Instruction-Driven Game Engine: A Poker Case Study](https://mlsanity.com/paper/2410.13441/) (2024)
- [Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis](https://mlsanity.com/paper/2505.12462/) (2025)
- [What-If Analysis of Large Language Models: Explore the Game World Using Proactive Thinking](https://mlsanity.com/paper/2509.04791/) (2025)
- [Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules](https://mlsanity.com/paper/2306.17766/) (2023)
- [HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making](https://mlsanity.com/paper/2509.12927/) (2025)
- [Offline Fictitious Self-Play for Competitive Games](https://mlsanity.com/paper/2403.00841/) (2024)
- [Stimulate the Potential of Robots via Competition](https://mlsanity.com/paper/2403.10487/) (2024)
- [HistoGym: A Reinforcement Learning Environment for Histopathological Image Analysis](https://mlsanity.com/paper/2408.08847/) (2024)
- [Mxplainer: Explain and Learn Insights by Imitating Mahjong Agents](https://mlsanity.com/paper/2506.14246/) (2025)
- [Mini Honor of Kings: A Lightweight Environment for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.03978/) (2024)
- [A Practical Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/2505.08295/) (2025)
- [Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control](https://mlsanity.com/paper/2507.15174/) (2025)
- [Augmenting Game AI with Deep Reinforcement Learning](https://mlsanity.com/paper/2606.20210/) (2026)
- [MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL](https://mlsanity.com/paper/2610.14883/) (2026)

## Abstract (arXiv)

This paper introduces Honor of Kings Arena, a reinforcement learning (RL) environment based on Honor of Kings, one of the world's most popular games at present. Compared to other environments studied in most previous work, ours presents new generalization challenges for competitive reinforcement learning. It is a multi-agent problem with one agent competing against its opponent; and it requires the generalization ability as it has diverse targets to control and diverse opponents to compete with. We describe the observation, action, and reward specifications for the Honor of Kings domain and provide an open-source Python-based interface for communicating with the game engine. We provide twenty target heroes with a variety of tasks in Honor of Kings Arena and present initial baseline results for RL-based methods with feasible computing resources. Finally, we showcase the generalization challenges imposed by Honor of Kings Arena and possible remedies to the challenges. All of the software, including the environment-class, are publicly available at https://github.com/tencent-ailab/hok_env . The documentation is available at https://aiarena.tencent.com/hok/doc/ .