mlsanityData to 7 Oct 2026

ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

arXiv 2410.01954 ↗First posted 2 Oct 2024cs.LG cs.MAcited by 8

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi-agent value decomposition strategy to facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 62 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

51 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation3 Nov
  2. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  3. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  4. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 47

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
35 more
  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  2. 2019The StarCraft Multi-Agent Challengecited by 589
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  5. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  8. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  10. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  11. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  12. 2021Offline RL Without Off-Policy Evaluationcited by 150
  13. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  14. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  15. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  16. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  17. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  18. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  19. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 94
  20. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  21. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  22. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  23. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  24. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  25. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  26. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  27. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  28. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  29. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  30. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  31. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  32. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 18
  33. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  34. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learningcited by 16
  35. 2023Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  2. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  3. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  4. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learningcited by 2
  5. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  6. 2026Offline Multi-Agent Reinforcement Learning with a Physics-Informed World Model for Cooperative Mixed Traffic Controlcited by 0
  7. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  8. 2026MA-WAM: Multi-Agent World-Action Model for Test-Time Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  2. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  3. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL
  4. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning
  5. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization
  6. 2026CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning
  7. 2025Partial Action Replacement: Tackling Distribution Shift in Offline MARL
  8. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning