mlsanityData to 14 Oct 2026

Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning

arXiv 2609.03667 ↗First posted 3 Sep 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Generalising to unseen tasks remains a fundamental challenge in offline multi-agent reinforcement learning (MARL). In this work, we present a principled analysis of zero-shot task generalisation in the offline setting and conduct an extensive empirical investigation into the scaling behaviour governing task diversity, dataset size, and network capacity. To facilitate this study, we extend offline sequence modelling architectures to handle multi-task observation and action spaces alongside variable agent counts across tasks. Our primary finding is that scaling task diversity---rather than sheer dataset size is the dominant factor in achieving robust zero-shot transfer. Through large-scale experiments across four challenging environments (Connector, RWARE, SMAX, and LBF), we demonstrate that our multi-task approach achieves a mean improvement of 3.2x on held-out test tasks compared to single-task models and consistently outperforms strong behaviour cloning baselines. These results suggest that the development of generalisable MARL agents should prioritise the diversity of the training distribution with varying numbers of agents, providing a roadmap for scaling offline MARL effectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 63 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

49 of 63 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization10 Dec
  2. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration25 Oct
  3. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization2 Oct
  4. 2024Coordination Failure in Cooperative Offline MARL1 Jul
  5. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning13 Feb
  6. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation3 Nov
  7. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  8. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem26 May
  9. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  10. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 39

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  6. 2019The StarCraft Multi-Agent Challengecited by 597
  7. 2022Multi-Game Decision Transformerscited by 214
  8. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  9. 2022Online Decision Transformercited by 196
  10. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  11. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
27 more
  1. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  2. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
  3. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  4. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  5. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  6. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  7. 2018Improving Regression Performance with Distributional Lossescited by 67
  8. 2023Elastic Decision Transformercited by 67
  9. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  10. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  11. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58
  12. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
  13. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
  14. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44
  15. 2023Scaling laws for single-agent reinforcement learningcited by 41
  16. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 37
  17. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 33
  18. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 30
  19. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  20. 2020Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Controlcited by 28
  21. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  22. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  23. 2023Offline Learning in Markov Games with General Function Approximationcited by 18
  24. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  25. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  26. 2023MaskMA: Towards Zero-Shot Multi-Agent Decision Making with Mask-Based Collaborative Learningcited by 3
  27. 2024Multi-Agent Reinforcement Learning with Selective State-Space Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  2. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
  3. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling
  4. 2024Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning
  5. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning
  6. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
  7. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions
  8. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information