mlsanityData to 7 Oct 2026

Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL

arXiv 2307.01472 ↗First posted 4 Jul 2023cs.AI cs.LG cs.MAcited by 15

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

We present a novel Diffusion Offline Multi-agent Model (DOM2) for offline Multi-Agent Reinforcement Learning (MARL). Different from existing algorithms that rely mainly on conservatism in policy design, DOM2 enhances policy expressiveness and diversity based on diffusion model. Specifically, we incorporate a diffusion model into the policy network and propose a trajectory-based data-reweighting scheme in training. These key ingredients significantly improve algorithm robustness against environment changes and achieve significant improvements in performance, generalization and data-efficiency. Our extensive experimental results demonstrate that DOM2 outperforms existing state-of-the-art methods in all multi-agent particle and multi-agent MuJoCo environments, and generalizes significantly better to shifted environments (in 28 out of 30 settings evaluated) thanks to its high expressiveness and diversity. Moreover, DOM2 is ultra data efficient and requires no more than 5% data for achieving the same performance compared to existing algorithms (a 20× improvement in data efficiency).

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 56 of its 68 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

56 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  2. 2023MADiff: Offline Multi-agent Learning with Diffusion Models27 May
  3. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  7. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion8 Sep
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 48

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
36 more
  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2021Variational Diffusion Modelscited by 1,145
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  7. 2022Diffusion Models in Vision: A Surveycited by 1,008
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  13. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  14. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  15. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  16. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  17. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  18. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  19. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  20. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  21. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  22. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  23. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  24. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  25. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  26. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  27. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 171
  28. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  29. 2023Synthetic Experience Replaycited by 100
  30. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  31. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  32. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  33. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  34. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  35. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  36. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 5; 2026: 4.

15 papers.

Sort:

By area:

  1. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  2. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  3. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimizationcited by 14
  4. 2025Multi-agent Coordination via Flow Matchingcited by 9
  5. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  6. 2023Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learningcited by 4
  7. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  8. 2026Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learningcited by 3
  9. 2024Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learningcited by 1
  10. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  11. 2025Multi-Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Plannercited by 1
  12. 2025From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demoscited by 1
3 more
  1. 2026CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learningcited by 0
  2. 2026MA-WAM: Multi-Agent World-Action Model for Test-Time Planningcited by 0
  3. 2026MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies
  2. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning
  3. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  4. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
  5. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  6. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
  7. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps
  8. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition