# DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.06135 · HTML: https://mlsanity.com/paper/2106.06135/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Combining Deep Reinforcement Learning and Search for Imperfect-Information Games](https://mlsanity.com/paper/2007.13544/) (2020)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Deep Counterfactual Regret Minimization](https://mlsanity.com/paper/1811.00164/) (2018)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [Solving Imperfect-Information Games via Discounted Regret Minimization](https://mlsanity.com/paper/1809.04040/) (2018)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [Suphx: Mastering Mahjong with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.13590/) (2020)
- [Experience Replay Optimization](https://mlsanity.com/paper/1906.08387/) (2019)
- [TorchBeast: A PyTorch Platform for Distributed RL](https://mlsanity.com/paper/1910.03552/) (2019)
- [Human-Level Performance in No-Press Diplomacy via Equilibrium Search](https://mlsanity.com/paper/2010.02923/) (2020)
- [RLCard: A Toolkit for Reinforcement Learning in Card Games](https://mlsanity.com/paper/1910.04376/) (2019)
- [Improving Policies via Search in Cooperative Partially Observable Games](https://mlsanity.com/paper/1912.02318/) (2019)

## Built on by (most cited first, 30 of 66)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization](https://mlsanity.com/paper/2402.17574/) (2024)
- [Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4](https://mlsanity.com/paper/2309.17277/) (2023)
- [Search Self-play: Pushing the Frontier of Agent Capability without Supervision](https://mlsanity.com/paper/2510.18821/) (2025)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regret](https://mlsanity.com/paper/2206.04122/) (2022)
- [Neurosymbolic Reinforcement Learning and Planning: A Survey](https://mlsanity.com/paper/2309.01038/) (2023)
- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [AutoShard: Automated Embedding Table Sharding for Recommender Systems](https://mlsanity.com/paper/2208.06399/) (2022)
- [Variational Quantum Soft Actor-Critic](https://mlsanity.com/paper/2112.11921/) (2021)
- [PerfectDou: Dominating DouDizhu with Perfect Information Distillation](https://mlsanity.com/paper/2203.16406/) (2022)
- [DreamShard: Generalizable Embedding Table Placement for Recommender Systems](https://mlsanity.com/paper/2210.02023/) (2022)
- [Games for Artificial Intelligence Research: A Review and Perspectives](https://mlsanity.com/paper/2304.13269/) (2023)
- [Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.22995/) (2026)
- [Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL](https://mlsanity.com/paper/2207.05683/) (2022)
- [Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2208.12433/) (2022)
- [Concentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems](https://mlsanity.com/paper/2203.06416/) (2022)
- [DouZero+: Improving DouDizhu AI by Opponent Modeling and Coach-guided Learning](https://mlsanity.com/paper/2204.02558/) (2022)
- [AutoVideo: An Automated Video Action Recognition System](https://mlsanity.com/paper/2108.04212/) (2021)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [DanZero+: Dominating the GuanDan Game through Reinforcement Learning](https://mlsanity.com/paper/2312.02561/) (2023)
- [ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2508.03768/) (2025)
- [DanZero: Mastering GuanDan Game with Reinforcement Learning](https://mlsanity.com/paper/2210.17087/) (2022)
- [Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models](https://mlsanity.com/paper/2305.01868/) (2023)
- [An Integrated Framework Integrating Monte Carlo Tree Search and Supervised Learning for Train Timetabling Problem](https://mlsanity.com/paper/2311.00971/) (2023)
- [Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox](https://mlsanity.com/paper/2212.00253/) (2022)
- [Can Large Language Models Master Complex Card Games?](https://mlsanity.com/paper/2509.01328/) (2025)
- [A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games](https://mlsanity.com/paper/2207.08894/) (2022)
- [Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning](https://mlsanity.com/paper/2302.06271/) (2023)
- [Imitation Learning from Purified Demonstrations](https://mlsanity.com/paper/2310.07143/) (2023)

## Abstract (arXiv)

Games are abstractions of the real world, where artificial agents learn to compete and cooperate with other agents. While significant achievements have been made in various perfect- and imperfect-information games, DouDizhu (a.k.a. Fighting the Landlord), a three-player card game, is still unsolved. DouDizhu is a very challenging domain with competition, collaboration, imperfect information, large state space, and particularly a massive set of possible actions where the legal actions vary significantly from turn to turn. Unfortunately, modern reinforcement learning algorithms mainly focus on simple and small action spaces, and not surprisingly, are shown not to make satisfactory progress in DouDizhu. In this work, we propose a conceptually simple yet effective DouDizhu AI system, namely DouZero, which enhances traditional Monte-Carlo methods with deep neural networks, action encoding, and parallel actors. Starting from scratch in a single server with four GPUs, DouZero outperformed all the existing DouDizhu AI programs in days of training and was ranked the first in the Botzone leaderboard among 344 AI agents. Through building DouZero, we show that classic Monte-Carlo methods can be made to deliver strong results in a hard domain with a complex action space. The code and an online demo are released at https://github.com/kwai/DouZero with the hope that this insight could motivate future work.