mlsanityData to 14 Oct 2026

Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games

arXiv 2411.04976 ↗First posted 7 Nov 2024cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Zero-shot coordination (ZSC) is a popular setting for studying the ability of reinforcement learning (RL) agents to coordinate with novel partners. Prior ZSC formulations assume the textitproblem setting is common knowledge: each agent knows the underlying Dec-POMDP, knows others have this knowledge, and so on ad infinitum. However, this assumption rarely holds in complex real-world settings, which are often difficult to fully and correctly specify. Hence, in settings where this common knowledge assumption is invalid, agents trained using ZSC methods may not be able to coordinate well. To address this limitation, we formulate the textitnoisy zero-shot coordination (NZSC) problem. In NZSC, agents observe different noisy versions of the ground truth Dec-POMDP, which are assumed to be distributed according to a fixed noise model. Only the distribution of ground truth Dec-POMDPs and the noise model are common knowledge. We show that a NZSC problem can be reduced to a ZSC problem by designing a meta-Dec-POMDP with an augmented state space consisting of all the ground-truth Dec-POMDPs. For solving NZSC problems, we propose a simple and flexible meta-learning method called NZSC training, in which the agents are trained across a distribution of coordination problems - which they only get to observe noisy versions of. We show that with NZSC training, RL agents can be trained to coordinate well with novel partners even when the (exact) problem setting of the coordination is not common knowledge.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 54 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

37 of 54 references.

References · 37

  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  4. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  6. 2016Cooperative Inverse Reinforcement Learningcited by 300
  7. 2021Unsolved Problems in ML Safetycited by 292
  8. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  9. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  10. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  11. 2020Open Problems in Cooperative AIcited by 168
  12. 2021Collaborating with Humans without Human Datacited by 142
25 more
  1. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  2. 2022Robust Reinforcement Learning using Offline Datacited by 96
  3. 2021Off-Belief Learningcited by 75
  4. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  5. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  6. 2017Reinforcement Learning under Model Mismatchcited by 52
  7. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  8. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 48
  9. 2020Inverse Constrained Reinforcement Learningcited by 46
  10. 2022A Survey of Ad Hoc Teamwork Researchcited by 43
  11. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  12. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 36
  13. 2019Preferences Implicit in the State of the Worldcited by 34
  14. 2021A New Formalism, Method and Open Issues for Zero-Shot Coordinationcited by 33
  15. 2022K-level Reasoning for Zero-Shot Coordination in Hanabicited by 32
  16. 2023PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordinationcited by 25
  17. 2021Quasi-Equivalence Discovery for Zero-Shot Emergent Communicationcited by 19
  18. 2023Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Gamecited by 18
  19. 2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
  20. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  21. 2019Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Gamescited by 11
  22. 2023Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabicited by 10
  23. 2023Efficient Human-AI Coordination via Preparatory Language-based Conventioncited by 7
  24. 2020Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communicationcited by 6
  25. 2024Symmetry-Breaking Augmentations for Ad Hoc Teamworkcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
  2. 2024KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination
  3. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination
  4. 2025Efficient Reinforcement Learning for Zero-Shot Coordination in Evolving Games
  5. 2025On Zero-Shot Reinforcement Learning
  6. 2025A Unified Framework for Zero-Shot Reinforcement Learning
  7. 2026The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
  8. 2026Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings