mlsanityData to 7 Oct 2026

Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank

arXiv 2410.01101 ↗First posted 1 Oct 2024cs.LGcited by 6

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

We study the problem of learning an approximate equilibrium in the offline multi-agent reinforcement learning (MARL) setting. We introduce a structural assumption -- the interaction rank -- and establish that functions with low interaction rank are significantly more robust to distribution shift compared to general ones. Leveraging this observation, we demonstrate that utilizing function classes with low interaction rank, when combined with regularization and no-regret learning, admits decentralized, computationally and statistically efficient learning in offline MARL. Our theoretical results are complemented by experiments that showcase the potential of critic architectures with low interaction rank in offline MARL, contrasting with commonly used single-agent value decomposition architectures.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 51 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

30 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization21 Jul
  2. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 28

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  7. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  9. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  10. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  11. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  12. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
16 more
  1. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  2. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  3. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  4. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  5. 2022The Complexity of Markov Equilibrium in Stochastic Gamescited by 45
  6. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  7. 2020Multi-Agent Reinforcement Learning in Stochastic Networked Systemscited by 30
  8. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  9. 2022Regret Minimization and Convergence to Equilibria in General-sum Markov Gamescited by 22
  10. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  11. 2023Multi-Player Zero-Sum Markov Games with Networked Separable Interactionscited by 8
  12. 2023Zero-sum Polymatrix Markov Games: Equilibrium Collapse and Efficient Computation of Nash Equilibriacited by 5
  13. 2024Locally Interdependent Multi-Agent MDP: Theoretical Framework for Decentralized Agents with Dynamic Dependenciescited by 5
  14. 2023Markov Games with Decoupled Dynamics: Price of Anarchy and Sample Complexitycited by 3
  15. 2023Guarantees for Self-Play in Multiplayer Games via Polymatrix Decomposabilitycited by 3
  16. 2024Approximate Global Convergence of Independent Learning in Multi-Agent Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debatecited by 7
  2. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learningcited by 2
  3. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  4. 2026Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representationscited by 0
  5. 2026G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policiescited by 0
  6. 2026MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
  2. 2026A Recipe for Stable Offline Multi-agent Reinforcement Learning
  3. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
  4. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  5. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  6. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
  7. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  8. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification