mlsanityData to 7 Oct 2026

Offline Decentralized Multi-Agent Reinforcement Learning

arXiv 2108.01832 ↗First posted 4 Aug 2021cs.LG cs.MAcited by 32

Not on the map. Why?

Cited by 27 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many real-world multi-agent cooperative tasks, due to high cost and risk, agents cannot continuously interact with the environment and collect experiences during learning, but have to learn from offline datasets. However, the transition dynamics in the dataset of each agent can be much different from the ones induced by the learned policies of other agents in execution, creating large errors in value estimates. Consequently, agents learn uncoordinated low-performing policies. In this paper, we propose a framework for offline decentralized multi-agent reinforcement learning, which exploits value deviation and transition normalization to deliberately modify the transition probabilities. Value deviation optimistically increases the transition probabilities of high-value next states, and transition normalization normalizes the transition probabilities of next states. They together enable agents to learn high-performing and coordinated policies. Theoretically, we prove the convergence of Q-learning under the altered non-stationary transition dynamics. Empirically, we show that the framework can be easily built on many existing offline reinforcement learning algorithms and achieve substantial improvement in a variety of multi-agent tasks.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

30 of 48 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016OpenAI Gymcited by 1,684
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
18 more
  1. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2019The StarCraft Multi-Agent Challengecited by 589
  6. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  10. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  11. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  12. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  13. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  14. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  15. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  16. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  17. 2020Learning Individually Inferred Communication for Multi-Agent Cooperationcited by 55
  18. 2017Lenient Multi-Agent Deep Reinforcement Learningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 8; 2024: 7; 2025: 5; 2026: 4.

32 papers.

Sort:

By area:

  1. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  2. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  3. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  4. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  5. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  6. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  7. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  8. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  9. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 18
  10. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  11. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  12. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
20 more
  1. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  2. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  3. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  4. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  5. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  6. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 7
  7. 2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
  8. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  9. 2022DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matchingcited by 5
  10. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  11. 2024Select to Perfect: Imitating desired behavior from large multi-agent datacited by 4
  12. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  13. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  14. 2023Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learningcited by 2
  15. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  16. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  17. 2026A Recipe for Stable Offline Multi-agent Reinforcement Learningcited by 0
  18. 2026CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learningcited by 0
  19. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Compositioncited by 0
  20. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  2. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization
  3. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  6. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning
  7. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  8. 2023Offline Learning in Markov Games with General Function Approximation