# Mastering Strategy Card Game (Hearthstone) with Improved Techniques

arXiv: https://arxiv.org/abs/2303.05197 · HTML: https://mlsanity.com/paper/2303.05197/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Deep Counterfactual Regret Minimization](https://mlsanity.com/paper/1811.00164/) (2018)
- [Learning in games with continuous action sets and unknown payoff functions](https://mlsanity.com/paper/1608.07310/) (2016)
- [A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games](https://mlsanity.com/paper/2206.05825/) (2022)
- [Suphx: Mastering Mahjong with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.13590/) (2020)
- [Unsupervised Predictive Memory in a Goal-Directed Agent](https://mlsanity.com/paper/1803.10760/) (2018)
- [Last-iterate Convergence in Extensive-Form Games](https://mlsanity.com/paper/2106.14326/) (2021)
- [Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms](https://mlsanity.com/paper/1808.04794/) (2018)
- [TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2011.12895/) (2020)
- [Introducing the Hearthstone-AI Competition](https://mlsanity.com/paper/1906.04238/) (2019)
- [Mastering Strategy Card Game (Legends of Code and Magic) via End-to-End Policy and Optimistic Smooth Fictitious Play](https://mlsanity.com/paper/2303.04096/) (2023)

## Built on by (most cited first, 10 of 10)

- [Contrastive Learning of Preferences with a Contextual InfoNCE Loss](https://mlsanity.com/paper/2407.05898/) (2024)
- [Summarizing Strategy Card Game AI Competition](https://mlsanity.com/paper/2305.11814/) (2023)
- [Learning With Generalised Card Representations for "Magic: The Gathering"](https://mlsanity.com/paper/2407.05879/) (2024)
- [Introducing Tales of Tribute AI Competition](https://mlsanity.com/paper/2305.08234/) (2023)
- [UrzaGPT: LoRA-Tuned Large Language Models for Card Selection in Collectible Card Games](https://mlsanity.com/paper/2508.08382/) (2025)
- [AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents](https://mlsanity.com/paper/2607.02255/) (2026)
- [Learning to Beat ByteRL: Exploitability of Collectible Card Game Agents](https://mlsanity.com/paper/2404.16689/) (2024)
- [Neural Network-based Information Set Weighting for Playing Reconnaissance Blind Chess](https://mlsanity.com/paper/2407.05864/) (2024)
- [Predicting Drafted Deck Strength for "Magic: the Gathering"](https://mlsanity.com/paper/2607.04782/) (2026)
- [A Taxonomy of Collectible Card Games from a Game-Playing AI Perspective](https://mlsanity.com/paper/2410.06299/) (2024)

## Abstract (arXiv)

Strategy card game is a well-known genre that is demanding on the intelligent game-play and can be an ideal test-bench for AI. Previous work combines an end-to-end policy function and an optimistic smooth fictitious play, which shows promising performances on the strategy card game Legend of Code and Magic. In this work, we apply such algorithms to Hearthstone, a famous commercial game that is more complicated in game rules and mechanisms. We further propose several improved techniques and consequently achieve significant progress. For a machine-vs-human test we invite a Hearthstone streamer whose best rank was top 10 of the official league in China region that is estimated to be of millions of players. Our models defeat the human player in all Best-of-5 tournaments of full games (including both deck building and battle), showing a strong capability of decision making.