mlsanityData to 7 Oct 2026

Off-Belief Learning

arXiv 2103.04000 ↗First posted 6 Mar 2021cs.AI cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

The standard problem setting in Dec-POMDPs is self-play, where the goal is to find a set of policies that play optimally together. Policies learned through self-play may adopt arbitrary conventions and implicitly rely on multi-step reasoning based on fragile assumptions about other agents' actions and thus fail when paired with humans or independently trained agents at test time. To address this, we present off-belief learning (OBL). At each timestep OBL agents follow a policy π1 that is optimized assuming past actions were taken by a given, fixed policy (π0), but assuming that future actions will be taken by π1. When π0 is uniform random, OBL converges to an optimal policy that does not rely on inferences based on other agents' behavior (an optimal grounded policy). OBL can be iterated in a hierarchy, where the optimal policy from one level becomes the input to the next, thereby introducing multi-level cognitive reasoning in a controlled manner. Unlike existing approaches, which may converge to any equilibrium policy, OBL converges to a unique policy, making it suitable for zero-shot coordination (ZSC). OBL can be scaled to high-dimensional settings with a fictitious transition mechanism and shows strong performance in both a toy-setting and the benchmark human-AI & ZSC problem Hanabi.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

14 of 29 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2015Prioritized Experience Replaycited by 1,070
  4. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  5. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  6. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  7. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  8. 2018Distributed Prioritized Experience Replaycited by 260
  9. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  10. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  11. 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 67
  12. 2019Simplified Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 47
2 more
  1. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  2. 2018Multi-Agent Common Knowledge Reinforcement Learningcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 16; 2023: 13; 2024: 8; 2025: 15; 2026: 15.

72 papers.

Sort:

By area:

  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  2. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  3. 2024Simplifying Deep Temporal Difference Learningcited by 71
  4. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 65
  5. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  6. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  7. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
  8. 2022A Survey of Ad Hoc Teamwork Researchcited by 41
  9. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  10. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  11. 2022K-level Reasoning for Zero-Shot Coordination in Hanabicited by 30
  12. 2022Any-Play: An Intrinsic Augmentation for Zero-Shot Coordinationcited by 22
38 more
  1. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  2. 2024Human-compatible driving partners through data-regularized self-play reinforcement learningcited by 17
  3. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  4. 2022Equivariant Networks for Zero-Shot Coordinationcited by 16
  5. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  6. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  7. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
  8. 2022Heterogeneous Multi-agent Zero-Shot Coordination by Coevolutioncited by 12
  9. 2022Mimetic Models: Ethical Implications of AI that Acts Like Youcited by 11
  10. 2021Learned Belief Search: Efficiently Improving Policies in Partially Observable Settingscited by 10
  11. 2023Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabicited by 10
  12. 2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
  13. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9
  14. 2022Human-AI Coordination via Human-Regularized Search and Learningcited by 8
  15. 2022On-the-fly Strategy Adaptation for ad-hoc Agent Coordinationcited by 7
  16. 2023A Hierarchical Approach to Population Training for Human-AI Collaborationcited by 7
  17. 2023Efficient Human-AI Coordination via Preparatory Language-based Conventioncited by 7
  18. 2023Cooperative AI via Decentralized Commitment Devicescited by 7
  19. 2025ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamworkcited by 6
  20. 2025Ad-Hoc Human-AI Coordination Challengecited by 6
  21. 2022On the Impossibility of Learning to Cooperate with Adaptive Partner Strategies in Repeated Gamescited by 5
  22. 2022Generalized Beliefs for Cooperative AIcited by 5
  23. 2025A Generalist Hanabi Agentcited by 5
  24. 2023Cheap Talk Discovery and Utilization in Multi-Agent Reinforcement Learningcited by 4
  25. 2025Communication-Efficient Desire Alignment for Embodied Agent-Human Adaptationcited by 4
  26. 2022Learning Intuitive Policies Using Action Featurescited by 3
  27. 2025Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Explorationcited by 3
  28. 2025The Yokai Learning Environment: Tracking Beliefs Over Space and Timecited by 3
  29. 2026CRAFT: Grounded Multi-Agent Coordination Under Partial Informationcited by 3
  30. 2021Attention Based Communication and Control for Multi-UAV Path Planningcited by 2
  31. 2023Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmascited by 2
  32. 2023Towards A Natural Language Interface for Flexible Multi-Agent Task Assignmentcited by 2
  33. 2024Symmetry-Breaking Augmentations for Ad Hoc Teamworkcited by 2
  34. 2025Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractionscited by 2
  35. 2025High entropy leads to symmetry-equivariant policies in Dec-POMDPscited by 2
  36. 2026Probing Dec-POMDP Reasoning in Cooperative MARLcited by 2
  37. 2026Training Generalizable Collaborative Agents via Strategic Risk Aversioncited by 2
  38. 2026Shaping Zero-Shot Coordination via State Blockingcited by 2

50 most cited of 72 All 72 (JSON)