mlsanityData to 7 Oct 2026

Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

arXiv 2602.18291 ↗First posted 20 Feb 2026cs.AIcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first underlineOnline off-policy underlineMARL framework using underlineDiffusion policies (OMAD) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across 10 diverse tasks, demonstrating a remarkable 2.5× to 5× improvement in sample efficiency.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 61 of its 81 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

61 of 81 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Enhanced DACER Algorithm with High Diffusion Efficiency29 May
  2. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective27 May
  3. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps15 May
  4. 2025Latent Diffusion Planning for Imitation Learning23 Apr
  5. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy17 Feb
  6. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning4 Feb
  7. 2025Efficient Online Reinforcement Learning for Diffusion Policy1 Feb
  8. 2024On Diffusion Models for Multi-Agent Partial Observability: Shared Attractors, Error Bounds, and Composite Flow17 Oct
  9. 2024Diffusion Policy Policy Optimization1 Sep
  10. 2024Beyond Local Views: Global State Inference with Diffusion Models for Cooperative Multi-Agent Reinforcement Learning18 Aug
  11. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  12. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
11 more
  1. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  2. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  3. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  4. 2023MADiff: Offline Multi-agent Learning with Diffusion Models27 May
  5. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  6. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  9. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  10. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  8. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 743
26 more
  1. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  5. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  6. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  7. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  8. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  9. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  10. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  11. 2023Synthetic Experience Replaycited by 100
  12. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  13. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  14. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 47
  15. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  16. 2025TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timescited by 31
  17. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  18. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  19. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  20. 2025Multi-Agent Systems for Robotic Autonomy with LLMscited by 9
  21. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learningcited by 4
  22. 2025Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Explorationcited by 3
  23. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  24. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  25. 2025Multi-Agent Reinforcement Learning with Communication-Constrained Priorscited by 3
  26. 2026GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Decentralized Master-Mind: Joint Action Refinement through Iterative Intent Denoising in Multi-Agent Pathfindingcited by 0
  2. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning
  2. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL
  3. 2026Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents
  4. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
  5. 2025OM2P: Offline Multi-Agent Mean-Flow Policy
  6. 2026Coordinated Diffusion: Generating Multi-Agent Behavior Without Multi-Agent Demonstrations
  7. 2025Training Diffusion Policies via Prior-Mapping Co-Evolution
  8. 2026CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning