mlsanityData to 7 Oct 2026

Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi

arXiv 2107.07630 ↗First posted 15 Jul 2021cs.AI cs.HCcited by 37

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no statistical difference in the game score. This result has implications for future AI design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-AI teaming rather than a singular focus on objective task performance.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 17 of its 53 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

17 of 53 references.

References · 17

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  3. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  4. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  5. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  6. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  7. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  8. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  9. 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 67
  10. 2019On the Pitfalls of Measuring Emergent Communicationcited by 66
  11. 2019Simplified Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 47
  12. 2020Human-Level Performance in No-Press Diplomacy via Equilibrium Searchcited by 41
5 more
  1. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  2. 2017Evaluating and Modelling Hanabi-Playing Agentscited by 9
  3. 2016Automatic Bridge Bidding Using Deep Reinforcement Learningcited by 6
  4. 2018Evolving Agents for the Hanabi 2018 CIG Competitioncited by 4
  5. 2019Re-determinizing Information Set Monte Carlo Tree Search in Hanabicited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 10; 2024: 5; 2025: 8; 2026: 5.

37 papers.

Sort:

By area:

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  3. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
  4. 2024Evaluating Human-AI Collaboration: A Review and Methodological Frameworkcited by 25
  5. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  6. 2022Any-Play: An Intrinsic Augmentation for Zero-Shot Coordinationcited by 22
  7. 2024Autonomous Industrial Control using an Agentic Framework with Large Language Modelscited by 19
  8. 2021Learning to run a power network with trustcited by 17
  9. 2023Diverse Conventions for Human-AI Collaborationcited by 14
  10. 2023Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Modelscited by 12
  11. 2022Warmth and competence in human-agent cooperationcited by 11
  12. 2023Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabicited by 10
25 more
  1. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  2. 2024Why Would You Suggest That? Human Trust in Language Model Responsescited by 9
  3. 2023Efficient Human-AI Coordination via Preparatory Language-based Conventioncited by 7
  4. 2022Interpretable Learned Emergent Communication for Human-Agent Teamscited by 6
  5. 2025Digital Player: Evaluating Large Language Models based Human-like Agent in Gamescited by 6
  6. 2023Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspectivecited by 5
  7. 2023Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goalscited by 4
  8. 2025Fewer Than 1% of Explainable AI Papers Validate Explainability with Humanscited by 4
  9. 2025Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulationcited by 3
  10. 2021Instructive artificial intelligence (AI) for human training, assistance, and explainabilitycited by 2
  11. 2023The Role of Heuristics and Biases During Complex Choices with an AI Teammatecited by 2
  12. 2024KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordinationcited by 2
  13. 2025X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learningcited by 2
  14. 2022Towards automating Codenames spymasters with deep reinforcement learningcited by 1
  15. 2025Human-AI Collaboration: Trade-offs Between Performance and Preferencescited by 1
  16. 2025In Pursuit of Predictive Models of Human Preferences Toward AI Teammatescited by 1
  17. 2025Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Reviewcited by 1
  18. 2021Reinforcement Learning on Human Decision Models for Uniquely Collaborative AI Teammatescited by 0
  19. 2024Towards Neural Network based Cognitive Models of Dynamic Decision-Making by Humanscited by 0
  20. 2025Moving Out: Physically-grounded Human-AI Collaborationcited by 0
  21. 2026Measuring Successful Cooperation in Human-AI Teamwork: Development and Validation of the Perceived Cooperativity and Teaming Perception Scalescited by 0
  22. 2026Beyond Partner Diversity: An Influence-Based Team Steering Framework for Zero-Shot Human-Machine Teamingcited by 0
  23. 2026Communicating Chess Strategies in Natural Languagecited by 0
  24. 2026AI Evaluation Should Work With Humanscited by 0
  25. 2026The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluationcited by 0