mlsanityData to 7 Oct 2026

Learning Partial Action Replacement in Offline MARL

arXiv 2603.28573 ↗First posted 30 Mar 2026cs.LG cs.AI cs.MAcited by 0

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Offline multi-agent reinforcement learning (MARL) faces a critical challenge: the joint action space grows exponentially with the number of agents, making dataset coverage exponentially sparse and out-of-distribution (OOD) joint actions unavoidable. Partial Action Replacement (PAR) mitigates this by anchoring a subset of agents to dataset actions, but existing approach relies on enumerating multiple subset configurations at high computational cost and cannot adapt to varying states. We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem and learns a state-dependent PAR policy using Proximal Policy Optimisation with an uncertainty-weighted reward. This adaptive policy dynamically determines how many agents to replace at each update step, balancing policy improvement against conservative value estimation. We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents. Compared with the previous PAR-based method SPaCQL, PLCQL reduces the number of per-iteration Q-function evaluations from n to 1, significantly improving computational efficiency. Empirically, PLCQL achieves the highest normalised scores on 66% of tasks across MPE, MaMuJoCo, and SMAC benchmarks, outperforming SPaCQL on 84% of tasks while substantially reducing computational cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 22 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

19 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Partial Action Replacement: Tackling Distribution Shift in Offline MARL10 Nov
  2. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  3. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2019The StarCraft Multi-Agent Challengecited by 589
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
4 more
  1. 2017Automated Curriculum Learning for Neural Networkscited by 251
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  3. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  4. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
  2. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  3. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning
  4. 2020MOReL : Model-Based Offline Reinforcement Learning
  5. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
  6. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition
  7. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  8. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL