mlsanityData to 7 Oct 2026

Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition

arXiv 2606.21085 ↗First posted 19 Jun 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Offline-to-online adaptation serves as a pivotal paradigm for mitigating the prohibitive cost of online exploration by bootstrapping reinforcement learning from offline datasets. While this paradigm has been extensively studied in single-agent settings, its extension to Multi-Agent Reinforcement Learning (MARL) remains largely unexplored, despite its critical relevance to complex coordinated decision-making. To bridge this gap, we introduce Sim2O, an elegant and minimalist framework for offline-to-online MARL. Rather than treating adaptation as a monolithic joint decision, Sim2O conceptualizes it as a compositional process. Specifically, candidate joint actions are synthesized by dynamically blending offline and online action proposals across agents. By leveraging a centralized value function to evaluate these hybrid combinations, Sim2O identifies high-value coordination strategies without requiring auxiliary training objectives or structural overhead. Empirical evaluations across diverse benchmarks demonstrate that Sim2O significantly outperforms existing baselines, underscoring that a minimalist design is not only viable but highly effective for multi-agent offline-to-online adaptation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 33 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

25 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  2. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  3. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 22

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
10 more
  1. 2020Critic Regularized Regressioncited by 269
  2. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  3. 2019Batch Policy Learning under Constraintscited by 242
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  5. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  6. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  7. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  8. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  9. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  10. 2026GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent Systemscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
  2. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration
  3. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL
  4. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning
  5. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  6. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
  7. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps
  8. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem