mlsanityData to 7 Oct 2026

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

arXiv 2111.11188 ↗First posted 22 Nov 2021cs.LGcited by 59

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, as many real-world scenarios involve interaction among multiple agents, it is important to resolve offline RL in the multi-agent setting. Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that offline RL algorithms will also transfer to multi-agent settings directly. Surprisingly, we empirically observe that conservative offline RL algorithms do not work well in the multi-agent setting -- the performance degrades significantly with an increasing number of agents. Towards mitigating the degradation, we identify a key issue that non-concavity of the value function makes the policy gradient improvements prone to local optima. Multiple agents exacerbate the problem severely, since the suboptimal policy by any agent can lead to uncoordinated global failure. Following this intuition, we propose a simple yet effective method, Offline Multi-Agent RL with Actor Rectification (OMAR), which combines the first-order policy gradients and zeroth-order optimization methods to better optimize the conservative value functions over the actor parameters. Despite the simplicity, OMAR achieves state-of-the-art results in a variety of multi-agent control tasks.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 69 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

54 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
38 more
  1. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  6. 2019The StarCraft Multi-Agent Challengecited by 589
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  9. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  10. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  11. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  12. 2020Critic Regularized Regressioncited by 269
  13. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  14. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  15. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  16. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  17. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  18. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  19. 2018Understanding the impact of entropy on policy optimizationcited by 171
  20. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  21. 2020Model-Based Offline Planningcited by 126
  22. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  23. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  24. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  25. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  26. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  27. 2020The Complexity of Constrained Min-Max Optimizationcited by 81
  28. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  29. 2018Evolution-Guided Policy Gradient in Reinforcement Learningcited by 78
  30. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  31. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  32. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 54
  33. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 53
  34. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  35. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 40
  36. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  37. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  38. 2019Q-Learning for Continuous Actions with Cross-Entropy Guided Policiescited by 17

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 13; 2025: 15; 2026: 17.

59 papers.

Sort:

By area:

  1. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  2. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  3. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  4. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  5. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  6. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  7. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  8. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 18
  9. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  10. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimationcited by 10
  11. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
  12. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
38 more
  1. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  2. 2025Multi-agent Coordination via Flow Matchingcited by 9
  3. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  4. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  5. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  6. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  7. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 7
  8. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  9. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  10. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  11. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actionscited by 5
  12. 2023Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learningcited by 4
  13. 2024Select to Perfect: Imitating desired behavior from large multi-agent datacited by 4
  14. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  15. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  16. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  17. 2026Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learningcited by 3
  18. 2023Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learningcited by 2
  19. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  20. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learningcited by 2
  21. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectificationcited by 2
  22. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  23. 2024ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Gamescited by 1
  24. 2024Diffusion Models for Offline Multi-agent Reinforcement Learning with Safety Constraintscited by 1
  25. 2024Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Trainingcited by 1
  26. 2025Offline Diffusion Policy for Multi-User Delay-Constrained Schedulingcited by 1
  27. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learningcited by 1
  28. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  29. 2025Multi-Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Plannercited by 1
  30. 2025From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demoscited by 1
  31. 2025Partial Action Replacement: Tackling Distribution Shift in Offline MARLcited by 1
  32. 2026STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learningcited by 1
  33. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0
  34. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Informationcited by 0
  35. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  36. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 0
  37. 2026A Recipe for Stable Offline Multi-agent Reinforcement Learningcited by 0
  38. 2026Learning Partial Action Replacement in Offline MARLcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  2. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  3. 2021Offline RL Without Off-Policy Evaluation
  4. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  5. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  6. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning