mlsanityData to 7 Oct 2026

MADiff: Offline Multi-agent Learning with Diffusion Models

arXiv 2305.17330 ↗First posted 27 May 2023cs.AI cs.LGcited by 73

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task. Q-learning algorithms struggle with extrapolation errors in offline settings, while supervised learning methods are constrained by model expressiveness. Recently, diffusion models (DMs) have shown promise in overcoming these limitations in single-agent learning, but their application in multi-agent scenarios remains unclear. Generating trajectories for each agent with independent DMs may impede coordination, while concatenating all agents' information can lead to low sample efficiency. Accordingly, we propose MADiff, which is realized with an attention-based diffusion model to model the complex coordination among behaviors of multiple agents. To our knowledge, MADiff is the first diffusion-based multi-agent learning framework, functioning as both a decentralized policy and a centralized controller. During decentralized executions, MADiff simultaneously performs teammate modeling, and the centralized controller can also be applied in multi-agent trajectory predictions. Our experiments demonstrate that MADiff outperforms baseline algorithms across various multi-agent learning tasks, highlighting its effectiveness in modeling complex multi-agent interactions. Our code is available at https://github.com/zbzhu99/madiff.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 45 of its 53 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

45 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  2. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
29 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2019The StarCraft Multi-Agent Challengecited by 589
  7. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  8. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  9. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  10. 2018Machine Theory of Mindcited by 222
  11. 2017Learning with Opponent-Learning Awarenesscited by 218
  12. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  13. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  14. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  15. 2023Synthetic Experience Replaycited by 100
  16. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  17. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  18. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 64
  19. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  20. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  21. 2020Agent Modelling under Partial Observability for Deep Reinforcement Learningcited by 56
  22. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  23. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  24. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  25. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  26. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  27. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  28. 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
  29. 2021baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agentscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 16; 2025: 28; 2026: 26.

73 papers.

Sort:

By area:

  1. 2023CDDM: Channel Denoising Diffusion Models for Wireless Semantic Communicationscited by 58
  2. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  3. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
  4. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  5. 2025Embodied AI: From LLMs to World Modelscited by 23
  6. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  7. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  8. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimizationcited by 14
  9. 2024Resisting Stochastic Risks in Diffusion Planners with the Trajectory Aggregation Treecited by 13
  10. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  11. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  12. 2025Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulationcited by 9
38 more
  1. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspectivecited by 9
  2. 2025Multi-agent Coordination via Flow Matchingcited by 9
  3. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  4. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
  5. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 7
  6. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debatecited by 7
  7. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  8. 2024Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replaycited by 6
  9. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timestepscited by 6
  10. 2025MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policiescited by 6
  11. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  12. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  13. 2024DiffLight: A Partial Rewards Conditioned Diffusion Model for Traffic Signal Control with Missing Datacited by 4
  14. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  15. 2025Adversarial Diffusion for Robust Reinforcement Learningcited by 4
  16. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Explorationcited by 3
  17. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  18. 2026Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learningcited by 3
  19. 2026Simulation-Informed Diffusion for Decentralized Multi-robot Motion Planningcited by 3
  20. 2023Diffusion Models for Multi-target Adversarial Trackingcited by 2
  21. 2024DyDiff: Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learningcited by 2
  22. 2025Diffusion-Guided Multi-Arm Motion Planningcited by 2
  23. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Developmentcited by 2
  24. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learningcited by 2
  25. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 2
  26. 2026Compositional Visual Planning via Inference-Time Diffusion Scalingcited by 2
  27. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  28. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  29. 2024Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learningcited by 1
  30. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learningcited by 1
  31. 2025RADE: Learning Risk-Adjustable Driving Environment via Multi-Agent Conditional Diffusioncited by 1
  32. 2025Scalable Multi Agent Diffusion Policies for Coverage Controlcited by 1
  33. 2025Actor-Critic without Actorcited by 1
  34. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 1
  35. 2025Multi-Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Plannercited by 1
  36. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learningcited by 1
  37. 2026Contractive Diffusion Policies: Robust Action Diffusion via Contractive Score-Based Sampling with Differential Equationscited by 1
  38. 2026Diffusion Integrated Gradients: Controllable Path Generation for Flexible Feature Attributioncited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Local Views: Global State Inference with Diffusion Models for Cooperative Multi-Agent Reinforcement Learning
  2. 2026Coordinated Diffusion: Generating Multi-Agent Behavior Without Multi-Agent Demonstrations
  3. 2024Diffusion Models for Offline Multi-agent Reinforcement Learning with Safety Constraints
  4. 2024ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories
  5. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  6. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
  7. 2026Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents
  8. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition