mlsanityData to 14 Oct 2026

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

arXiv 2309.12696 ↗First posted 22 Sep 2023cs.AIcited by 37

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. Tomitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 55 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

44 of 55 references.

Closest ancestors · 18

Same direction, newest first

  1. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  2. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  5. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
6 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2019The StarCraft Multi-Agent Challengecited by 597
  12. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
14 more
  1. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  2. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
  3. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  4. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  5. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  6. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  7. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  8. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  9. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  10. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
  11. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 47
  12. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 33
  13. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  14. 2022Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 13; 2026: 11.

37 papers.

Sort:

By area:

  1. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  2. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  3. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  4. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 11
  5. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 11
  6. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
  7. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  8. 2024Language-Conditioned Offline RL for Multi-Robot Navigationcited by 9
  9. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  10. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  11. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 8
  12. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
25 more
  1. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actionscited by 6
  2. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  3. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 5
  4. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  5. 2024Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learningcited by 4
  6. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  7. 2025Focusing Influence Mechanism for Multi-Agent Reinforcement Learningcited by 2
  8. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  9. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 2
  10. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  11. 2026A Recipe for Stable Offline Multi-agent Reinforcement Learningcited by 2
  12. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  13. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 1
  14. 2025From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demoscited by 1
  15. 2026STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learningcited by 1
  16. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
  17. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  18. 2026Learning Partial Action Replacement in Offline MARLcited by 0
  19. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  20. 2026ISEP: Implicit Support Expansion for Offline Reinforcement Learning via Stochastic Policy Optimizationcited by 0
  21. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Compositioncited by 0
  22. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0
  23. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  24. 2026Improving LLM Collaboration via Multi-Agent Preference Learningcited by 0
  25. 2026Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  2. 2024Strategically Conservative Q-Learning
  3. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  4. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  5. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learning
  6. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  7. 2025Partial Action Replacement: Tackling Distribution Shift in Offline MARL
  8. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization