mlsanityData to 7 Oct 2026

Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL

arXiv 2505.22151 ↗First posted 28 May 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

A key challenge in offline multi-agent reinforcement learning (MARL) is achieving effective many-agent multi-step coordination in complex environments. In this work, we propose Oryx, a novel algorithm for offline cooperative MARL to directly address this challenge. Oryx adapts the recently proposed retention-based architecture Sable and combines it with a sequential form of implicit constraint Q-learning (ICQ), to develop a novel offline autoregressive policy update scheme. This allows Oryx to solve complex coordination challenges while maintaining temporal coherence over long trajectories. We evaluate Oryx across a diverse set of benchmarks from prior works -- SMAC, RWARE, and Multi-Agent MuJoCo -- covering tasks of both discrete and continuous control, varying in scale and difficulty. Oryx achieves state-of-the-art performance on more than 80% of the 65 tested datasets, outperforming prior offline MARL methods and demonstrating robust generalisation across domains with many agents and long horizons. Finally, we introduce new datasets to push the limits of many-agent coordination in offline MARL, and demonstrate Oryx's superior ability to scale effectively in such settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 62 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

49 of 62 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition9 May
  2. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization10 Dec
  3. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration25 Oct
  4. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization2 Oct
  5. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning18 Sep
  6. 2024Coordination Failure in Cooperative Offline MARL1 Jul
  7. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation13 Jun
  8. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning13 Feb
  9. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation3 Nov
  10. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  11. 2023MADiff: Offline Multi-agent Learning with Diffusion Models27 May
  12. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem26 May
3 more
  1. 2023Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning31 Mar
  2. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks6 Dec
  3. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  5. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  7. 2019The StarCraft Multi-Agent Challengecited by 589
  8. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
  9. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  10. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  11. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  12. 2019Behaviour Suite for Reinforcement Learningcited by 129
22 more
  1. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  2. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  3. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  4. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  5. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  6. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 52
  7. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  8. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  9. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  10. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  11. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  12. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  13. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  14. 2022Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workerscited by 28
  15. 2025Building reliable sim driving agents by scaling self-playcited by 20
  16. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  17. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  18. 2022Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARLcited by 12
  19. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  20. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  21. 2024Multi Agent Reinforcement Learning for Sequential Satellite Assignment Problemscited by 4
  22. 2024Multi-Agent Reinforcement Learning with Selective State-Space Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition
  2. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning
  3. 2026CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning
  4. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning
  5. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  6. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
  7. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank
  8. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning