mlsanityData to 14 Oct 2026

Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

arXiv 2505.11100 ↗First posted 16 May 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-play-based methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies' space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. In addition, BiDist operates as a concise and efficient solution without the need for the complex and costly storage of past policies. We provide both theoretical analysis and empirical evidence to support BiDist's effectiveness. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space. We also present comprehensive ablation studies to reinforce BiDist's effectiveness and key success factors. Source codes are available in the supplementary material.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 60 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

33 of 60 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  9. 2016The Option-Critic Architecturecited by 448
  10. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  12. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
21 more
  1. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  3. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  4. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  5. 2017Learning with Opponent-Learning Awarenesscited by 222
  6. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  7. 2021Collaborating with Humans without Human Datacited by 142
  8. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  9. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  10. 2018Generalization and Regularization in DQNcited by 118
  11. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  12. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  13. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  14. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  15. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  16. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  17. 2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translationcited by 50
  18. 2022Melting Pot 2.0cited by 40
  19. 2020R2-B2: Recursive Reasoning-Based Bayesian Optimization for No-Regret Learning in Gamescited by 11
  20. 2019A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learningcited by 10
  21. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play
  2. 2024A Survey on Self-play Methods in Reinforcement Learning
  3. 2024Multi-agent cooperation through learning-aware policy gradients
  4. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective
  5. 2026Building Multi-Task Agentic LLMs via Two-Phase Distillation
  6. 2024Reinforcing Competitive Multi-Agents for Playing 'So Long Sucker'
  7. 2024A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning ↗
  8. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents