# Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabi

arXiv: https://arxiv.org/abs/2203.11656 · HTML: https://mlsanity.com/paper/2203.11656/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1811.01458/) (2018)
- [Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1912.02288/) (2019)
- [Is the Policy Gradient a Gradient?](https://mlsanity.com/paper/1906.07073/) (2019)
- [Improving Policies via Search in Cooperative Partially Observable Games](https://mlsanity.com/paper/1912.02318/) (2019)
- [Scalable Online Planning via Reinforcement Learning Fine-Tuning](https://mlsanity.com/paper/2109.15316/) (2021)
- [Diverse Agents for Ad-Hoc Cooperation in Hanabi](https://mlsanity.com/paper/1907.03840/) (2019)

## Built on by (most cited first, 3 of 3)

- [Lattice Protein Folding with Variational Annealing](https://mlsanity.com/paper/2502.20632/) (2025)
- [Policy Gradient Algorithms with Monte Carlo Tree Learning for Non-Markov Decision Processes](https://mlsanity.com/paper/2206.01011/) (2022)
- [Two-Step Reinforcement Learning for Multistage Strategy Card Game](https://mlsanity.com/paper/2311.17305/) (2023)

## Abstract (arXiv)

In pursuit of enhanced multi-agent collaboration, we analyze several on-policy deep reinforcement learning algorithms in the recently published Hanabi benchmark. Our research suggests a perhaps counter-intuitive finding, where Proximal Policy Optimization (PPO) is outperformed by Vanilla Policy Gradient over multiple random seeds in a simplified environment of the multi-agent cooperative card game. In our analysis of this behavior we look into Hanabi-specific metrics and hypothesize a reason for PPO's plateau. In addition, we provide proofs for the maximum length of a perfect game (71 turns) and any game (89 turns). Our code can be found at: https://github.com/bramgrooten/DeepRL-for-Hanabi