mlsanityData to 7 Oct 2026

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

arXiv 2106.03400 ↗First posted 7 Jun 2021cs.AIcited by 78

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 62 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

43 of 62 references.

References · 43

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
31 more
  1. 2019The StarCraft Multi-Agent Challengecited by 589
  2. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  3. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  4. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2018Maximum a Posteriori Policy Optimisationcited by 318
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  12. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  13. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  14. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  15. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  16. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  17. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  18. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  19. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 96
  20. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  21. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  22. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  23. 2016Q(λ) with Off-Policy Correctionscited by 48
  24. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  25. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  26. 2018Randomized Value Functions via Multiplicative Normalizing Flowscited by 15
  27. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  28. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11
  29. 2018Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigationcited by 9
  30. 2020Adversarial Attacks and Detection on Reinforcement Learning-Based Interactive Recommender Systemscited by 9
  31. 2021Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 17; 2024: 15; 2025: 11; 2026: 17.

78 papers.

Sort:

By area:

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  2. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  3. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  4. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  5. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  6. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  7. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  8. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  9. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  10. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  11. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  12. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
38 more
  1. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  2. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  3. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 18
  4. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
  5. 2021Causal Multi-Agent Reinforcement Learning: Review and Open Problemscited by 16
  6. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
  7. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  8. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  9. 2021Mava: a research library for distributed multi-agent reinforcement learning in JAXcited by 14
  10. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  11. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  12. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  13. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  14. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  15. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimationcited by 10
  16. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
  17. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  18. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  19. 2024Language-Conditioned Offline RL for Multi-Robot Navigationcited by 9
  20. 2025Multi-agent Coordination via Flow Matchingcited by 9
  21. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  22. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  23. 2022Contextual Transformer for Offline Meta Reinforcement Learningcited by 7
  24. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  25. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  26. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 7
  27. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debatecited by 7
  28. 2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
  29. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  30. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  31. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rankcited by 6
  32. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  33. 2022State Advantage Weighting for Offline RLcited by 5
  34. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  35. 2023Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learningcited by 4
  36. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  37. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  38. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  2. 2025Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning
  3. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  4. 2025PIQL: Projective Implicit Q-Learning with Support Constraint for Offline Reinforcement Learning
  5. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
  6. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  7. 2021The Least Restriction for Offline Reinforcement Learning
  8. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning