# Combinational Q-Learning for Dou Di Zhu

arXiv: https://arxiv.org/abs/1901.08925 · HTML: https://mlsanity.com/paper/1901.08925/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2019.

## Built on by (most cited first, 6 of 6)

- [DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06135/) (2021)
- [PerfectDou: Dominating DouDizhu with Perfect Information Distillation](https://mlsanity.com/paper/2203.16406/) (2022)
- [UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models](https://mlsanity.com/paper/2406.16382/) (2024)
- [Can Large Language Models Master Complex Card Games?](https://mlsanity.com/paper/2509.01328/) (2025)
- [Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.12896/) (2022)
- [AlphaDou: High-Performance End-to-End Doudizhu AI Integrating Bidding](https://mlsanity.com/paper/2407.10279/) (2024)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has gained a lot of attention in recent years, and has been proven to be able to play Atari games and Go at or above human levels. However, those games are assumed to have a small fixed number of actions and could be trained with a simple CNN network. In this paper, we study a special class of Asian popular card games called Dou Di Zhu, in which two adversarial groups of agents must consider numerous card combinations at each time step, leading to huge number of actions. We propose a novel method to handle combinatorial actions, which we call combinational Q-learning (CQL). We employ a two-stage network to reduce action space and also leverage order-invariant max-pooling operations to extract relationships between primitive actions. Results show that our method prevails over state-of-the art methods like naive Q-learning and A3C. We develop an easy-to-use card game environments and train all agents adversarially from sractch, with only knowledge of game rules and verify that our agents are comparative to humans. Our code to reproduce all reported results will be available online.