mlsanityData to 7 Oct 2026

Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning

arXiv 2409.12001 ↗First posted 18 Sep 2024cs.LG cs.AI cs.MAcited by 4

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Offline multi-agent reinforcement learning (MARL) is an exciting direction of research that uses static datasets to find optimal control policies for multi-agent systems. Though the field is by definition data-driven, efforts have thus far neglected data in their drive to achieve state-of-the-art results. We first substantiate this claim by surveying the literature, showing how the majority of works generate their own datasets without consistent methodology and provide sparse information about the characteristics of these datasets. We then show why neglecting the nature of the data is problematic, through salient examples of how tightly algorithmic performance is coupled to the dataset used, necessitating a common foundation for experiments in the field. In response, we take a big step towards improving data usage and data awareness in offline MARL, with three key contributions: (1) a clear guideline for generating novel datasets; (2) a standardisation of over 80 existing datasets, hosted in a publicly available repository, using a consistent storage format and easy-to-use API; and (3) a suite of analysis tools that allow us to understand these datasets better, aiding further development.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 58 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

41 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Coordination Failure in Cooperative Offline MARL1 Jul
  2. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation13 Jun
  3. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation3 Nov
  4. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  5. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorization15 Jun
  6. 2023MADiff: Offline Multi-agent Learning with Diffusion Models27 May
  7. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem26 May
  8. 2023Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning31 Mar
  9. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning28 Nov
  10. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  11. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2016End to End Learning for Self-Driving Carscited by 1,000
  9. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  12. 2019The StarCraft Multi-Agent Challengecited by 589
18 more
  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  2. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  3. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  4. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  5. 2019CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenariocited by 114
  6. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 114
  7. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  8. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  9. 2019Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Criticscited by 46
  10. 2023A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environmentcited by 37
  11. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  12. 2020Flatland-RL : Multi-Agent Reinforcement Learning on Trainscited by 32
  13. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  14. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  15. 2020Multi-Agent Routing Value Iteration Networkcited by 17
  16. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  17. 2022Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidancecited by 6
  18. 2024An open source Multi-Agent Deep Reinforcement Learning Routing Simulator for satellite networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  2. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  3. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  4. 2026G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning
  2. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  3. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  4. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning
  5. 2022Leveraging Offline Data in Online Reinforcement Learning
  6. 2020Offline Reinforcement Learning Hands-On
  7. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  8. 2024Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization