mlsanityData to 14 Oct 2026

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

arXiv 2307.11620 ↗First posted 21 Jul 2023cs.LG cs.MAcited by 43

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline m ulti-agent RL algorithm with implicit global-to-local v alue regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 49 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

37 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization15 Jun
  2. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  3. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 34

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  12. 2019The StarCraft Multi-Agent Challengecited by 597
22 more
  1. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  2. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  3. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  5. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  6. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 177
  7. 2021Offline RL Without Off-Policy Evaluationcited by 151
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  9. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  10. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  11. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  12. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  13. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  14. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  15. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  16. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 33
  17. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  18. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  19. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  20. 2022Imitating Past Successes can be Very Suboptimalcited by 23
  21. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  22. 2021Divergence-Regularized Multi-Agent Actor-Criticcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 13; 2026: 18.

43 papers.

Sort:

By area:

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  2. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  4. 2025Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignmentcited by 18
  5. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
  6. 2025Multi-agent Coordination via Flow Matchingcited by 10
  7. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  8. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  9. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  10. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  11. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rankcited by 6
  12. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
31 more
  1. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actionscited by 6
  2. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  3. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  4. 2024Adversarially Robust Decision Transformercited by 4
  5. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  6. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  7. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  8. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  9. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  10. 2026Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learningcited by 3
  11. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  12. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learningcited by 2
  13. 2026A Recipe for Stable Offline Multi-agent Reinforcement Learningcited by 2
  14. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  15. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learningcited by 1
  16. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 1
  17. 2025Partial Action Replacement: Tackling Distribution Shift in Offline MARLcited by 1
  18. 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learningcited by 1
  19. 2026STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learningcited by 1
  20. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learningcited by 1
  21. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0
  22. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  23. 2026DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Makingcited by 0
  24. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Compositioncited by 0
  25. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  26. 2026Offline Multi-agent Continual Cooperation via Skill Partition and Reusecited by 0
  27. 2026AERIS: Offline Policy Improvement for Multi-UAV Integrated Sensing and Communicationcited by 0
  28. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  29. 2026Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformerscited by 0
  30. 2026MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimizationcited by 0
  31. 2026Test-time Multi-agent Coordination by Decomposed Value Gradient Flowcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  2. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  3. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  4. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  5. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL
  6. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
  7. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration
  8. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions