mlsanityData to 7 Oct 2026

B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning

arXiv 2501.18138 ↗First posted 30 Jan 2025cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC) regularization to existing online RL algorithms -- has been shown to be effective; however, this approach is understudied in multi-agent settings. In particular, overestimation becomes worse in multi-agent settings due to the presence of multiple actions, resulting in the BC regularization-based approach easily suffering from either over-regularization or critic divergence. To address this, we propose a simple yet effective method, Behavior Cloning regularization with Critic Clipping (B3C), which clips the target critic value in policy evaluation based on the maximum return in the dataset and pushes the limit of the weight on the RL objective over BC regularization, thereby improving performance. Additionally, we leverage existing value factorization techniques, particularly non-linear factorization, which is understudied in offline settings. Integrated with non-linear value factorization, B3C outperforms state-of-the-art algorithms on various offline multi-agent benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 36 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

27 of 36 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learning3 Feb
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  6. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  7. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  8. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  9. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  10. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  11. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  12. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
7 more
  1. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  2. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  3. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  4. 2020Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorizationcited by 26
  5. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  6. 2023A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learningcited by 5
  7. 2024On Stateful Value Factorization in Multi-Agent Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning
  2. 2022TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets
  3. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
  4. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning
  5. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  6. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  7. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  8. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization