mlsanityData to 7 Oct 2026

A Recipe for Stable Offline Multi-agent Reinforcement Learning

arXiv 2603.08399 ↗First posted 9 Mar 2026cs.LG cs.AI cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Despite remarkable achievements in single-agent offline reinforcement learning (RL), multi-agent RL (MARL) has struggled to adopt this paradigm, largely persisting with on-policy training and self-play from scratch. One reason for this gap comes from the instability of non-linear value decomposition, leading prior works to avoid complex mixing networks in favor of linear value decomposition (e.g., VDN) with value regularization used in single-agent setups. In this work, we analyze the source of instability in non-linear value decomposition within the offline MARL setting. Our observations confirm that they induce value-scale amplification and unstable optimization. To alleviate this, we propose a simple technique, scale-invariant value normalization (SVN), that stabilizes actor-critic training without altering the Bellman fixed point. Empirically, we examine the interaction among key components of offline MARL (e.g., value decomposition, value learning, and policy extraction) and derive a practical recipe that unlocks its full potential.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 76 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

58 of 76 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025OM2P: Offline Multi-Agent Mean-Flow Policy8 Aug
  2. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization10 Dec
  3. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation13 Jun
  4. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning13 Feb
  5. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation3 Nov
  6. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  7. 2023MADiff: Offline Multi-agent Learning with Diffusion Models27 May
  8. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning1 Feb
  9. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  10. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2019The StarCraft Multi-Agent Challengecited by 589
36 more
  1. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  2. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  3. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  6. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  8. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  9. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  10. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
  11. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  12. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  13. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  14. 2019Deep Coordination Graphscited by 96
  15. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  16. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  17. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  18. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  19. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 54
  20. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  21. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  22. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  23. 2024POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfindingcited by 20
  24. 2024AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Datasetcited by 14
  25. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
  26. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  27. 2022Non-Linear Coordination Graphscited by 10
  28. 2023Scalable Multi-agent Covering Option Discovery based on Kronecker Graphscited by 9
  29. 2024On Centralized Critics in Multi-Agent Reinforcement Learningcited by 9
  30. 2025Multi-agent Coordination via Flow Matchingcited by 9
  31. 2020Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graphcited by 7
  32. 2025SocialJax: An Evaluation Suite for Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 7
  33. 2025Safe Multi-Agent Navigation guided by Goal-Conditioned Safe Reinforcement Learningcited by 5
  34. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 4
  35. 2025Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscalecited by 2
  36. 2025Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Corecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization
  2. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank
  3. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  4. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
  5. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning
  6. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
  7. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  8. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions