mlsanityData to 13 Oct 2026

Partial Action Replacement: Tackling Distribution Shift in Offline MARL

arXiv 2511.07629 ↗First posted 10 Nov 2025cs.LG cs.AI cs.MAcited by 1

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Offline multi-agent reinforcement learning (MARL) is severely hampered by the challenge of evaluating out-of-distribution (OOD) joint actions. Our core finding is that when the behavior policy is factorized - a common scenario where agents act fully or partially independently during data collection - a strategy of partial action replacement (PAR) can significantly mitigate this challenge. PAR updates a single or part of agents' actions while the others remain fixed to the behavioral data, reducing distribution shift compared to full joint-action updates. Based on this insight, we develop Soft-Partial Conservative Q-Learning (SPaCQL), using PAR to mitigate OOD issue and dynamically weighting different PAR strategies based on the uncertainty of value estimation. We provide a rigorous theoretical foundation for this approach, proving that under factorized behavior policies, the induced distribution shift scales linearly with the number of deviating agents rather than exponentially with the joint-action space. This yields a provably tighter value error bound for this important class of offline MARL problems. Our theoretical results also indicate that SPaCQL adaptively addresses distribution shift using uncertainty-informed weights. Our empirical results demonstrate SPaCQL enables more effective policy learning, and manifest its remarkable superiority over baseline algorithms when the offline dataset exhibits the independence structure.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 23 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

14 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  2. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization15 Jun
  3. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem26 May
  4. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov

Other references · 10

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,782
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 884
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  5. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 700
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 654
  7. 2016Deep Exploration via Bootstrapped DQNcited by 520
  8. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 374
  9. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  10. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning Partial Action Replacement in Offline MARLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  2. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  4. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
  5. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
  6. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
  7. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  8. 2024Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learning