# Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi

arXiv: https://arxiv.org/abs/2107.07630 · HTML: https://mlsanity.com/paper/2107.07630/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Emergent Complexity via Multi-Agent Competition](https://mlsanity.com/paper/1710.03748/) (2017)
- [On the Pitfalls of Measuring Emergent Communication](https://mlsanity.com/paper/1903.05168/) (2019)
- [Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1811.01458/) (2018)
- [Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1912.02288/) (2019)
- [Human-Level Performance in No-Press Diplomacy via Equilibrium Search](https://mlsanity.com/paper/2010.02923/) (2020)
- [Improving Policies via Search in Cooperative Partially Observable Games](https://mlsanity.com/paper/1912.02318/) (2019)
- [Evaluating and Modelling Hanabi-Playing Agents](https://mlsanity.com/paper/1704.07069/) (2017)
- [Automatic Bridge Bidding Using Deep Reinforcement Learning](https://mlsanity.com/paper/1607.03290/) (2016)
- [Evolving Agents for the Hanabi 2018 CIG Competition](https://mlsanity.com/paper/1809.09764/) (2018)
- [Re-determinizing Information Set Monte Carlo Tree Search in Hanabi](https://mlsanity.com/paper/1902.06075/) (2019)

## Built on by (most cited first, 30 of 40)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning](https://mlsanity.com/paper/2210.05492/) (2022)
- [Evaluating Human-AI Collaboration: A Review and Methodological Framework](https://mlsanity.com/paper/2407.19098/) (2024)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination](https://mlsanity.com/paper/2201.12436/) (2022)
- [Autonomous Industrial Control using an Agentic Framework with Large Language Models](https://mlsanity.com/paper/2411.05904/) (2024)
- [Learning to run a power network with trust](https://mlsanity.com/paper/2110.12908/) (2021)
- [Diverse Conventions for Human-AI Collaboration](https://mlsanity.com/paper/2310.15414/) (2023)
- [Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models](https://mlsanity.com/paper/2311.05720/) (2023)
- [Warmth and competence in human-agent cooperation](https://mlsanity.com/paper/2201.13448/) (2022)
- [Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi](https://mlsanity.com/paper/2308.10284/) (2023)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Why Would You Suggest That? Human Trust in Language Model Responses](https://mlsanity.com/paper/2406.02018/) (2024)
- [Efficient Human-AI Coordination via Preparatory Language-based Convention](https://mlsanity.com/paper/2311.00416/) (2023)
- [Interpretable Learned Emergent Communication for Human-Agent Teams](https://mlsanity.com/paper/2201.07452/) (2022)
- [Digital Player: Evaluating Large Language Models based Human-like Agent in Games](https://mlsanity.com/paper/2502.20807/) (2025)
- [Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspective](https://mlsanity.com/paper/2304.11632/) (2023)
- [Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goals](https://mlsanity.com/paper/2302.06609/) (2023)
- [Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans](https://mlsanity.com/paper/2503.16507/) (2025)
- [Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulation](https://mlsanity.com/paper/2502.18145/) (2025)
- [X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning](https://mlsanity.com/paper/2510.19150/) (2025)
- [Instructive artificial intelligence (AI) for human training, assistance, and explainability](https://mlsanity.com/paper/2111.01726/) (2021)
- [The Role of Heuristics and Biases During Complex Choices with an AI Teammate](https://mlsanity.com/paper/2301.05969/) (2023)
- [KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination](https://mlsanity.com/paper/2408.04336/) (2024)
- [Towards automating Codenames spymasters with deep reinforcement learning](https://mlsanity.com/paper/2212.14104/) (2022)
- [Human-AI Collaboration: Trade-offs Between Performance and Preferences](https://mlsanity.com/paper/2503.00248/) (2025)
- [In Pursuit of Predictive Models of Human Preferences Toward AI Teammates](https://mlsanity.com/paper/2503.15516/) (2025)
- [Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review](https://mlsanity.com/paper/2503.19607/) (2025)
- [The Convention Gap: Beyond Literal Content in Human-AI Cooperation](https://mlsanity.com/paper/2609.11489/) (2026)

## Abstract (arXiv)

Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no statistical difference in the game score. This result has implications for future AI design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-AI teaming rather than a singular focus on objective task performance.