mlsanityData to 14 Oct 2026

Multi-Agent Reinforcement Learning is a Sequence Modeling Problem

arXiv 2205.14953 ↗First posted 30 May 2022cs.MA cs.LGcited by 155

Not on the map. Why?

Abstract (arXiv)

Large sequence model (SM) such as GPT series and BERT has displayed outstanding performance and generalization capabilities on vision, language, and recently reinforcement learning tasks. A natural follow-up question is how to abstract multi-agent decision making into an SM problem and benefit from the prosperous development of SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the task is to map agents' observation sequence to agents' optimal action sequence. Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL. Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multi-agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such as Decision Transformer fit only pre-collected offline data, MAT is trained by online trials and errors from the environment in an on-policy fashion. To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks. Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents. See our project page at https://sites.google.com/view/multi-agent-transformer.

Key facts. Cited by 155 papers in the map (data through 14 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

34 of 51 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
22 more
  1. 2022A Generalist Agentcited by 808
  2. 2021Going deeper with Image Transformerscited by 764
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  4. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  6. 2019The StarCraft Multi-Agent Challengecited by 597
  7. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  10. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  11. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  12. 2019MAVEN: Multi-Agent Variational Explorationcited by 185
  13. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  14. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 94
  15. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  16. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 56
  17. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  18. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  19. 2020Multi-Agent Determinantal Q-Learningcited by 36
  20. 2019Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoningcited by 24
  21. 2021On the Complexity of Computing Markov Perfect Equilibrium in General-Sum Stochastic Gamescited by 19
  22. 2021LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 31; 2024: 41; 2025: 41; 2026: 37.

155 papers.

Sort:

By area:

  1. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  2. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 98
  3. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  4. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 75
  5. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 68
  6. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 52
  7. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  8. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 49
  9. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 48
  10. 2025Multi-Agent Coordination across Diverse Applications: A Surveycited by 37
  11. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  12. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
38 more
  1. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  2. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  3. 2025Embodied AI: From LLMs to World Modelscited by 24
  4. 2023TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Playcited by 20
  5. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  6. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  7. 2024DPN: Decoupling Partition and Navigation for Neural Solvers of Min-max Vehicle Routing Problemscited by 18
  8. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
  9. 2022A Survey on Influence Maximization: From an ML-Based Combinatorial Optimizationcited by 15
  10. 2022Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformercited by 14
  11. 2025Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Surveycited by 14
  12. 2025Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environmentscited by 14
  13. 2024Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Taskscited by 13
  14. 2024Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challengecited by 13
  15. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  16. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  17. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  18. 2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 11
  19. 2023A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decisioncited by 11
  20. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimationcited by 10
  21. 2024Large Language Models as Agents in Two-Player Gamescited by 10
  22. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  23. 2025Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learningcited by 10
  24. 2025Multi-agent Coordination via Flow Matchingcited by 10
  25. 2023IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARLcited by 9
  26. 2024Multi-Sender Persuasion: A Computational Perspectivecited by 9
  27. 2022On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspectivecited by 8
  28. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  29. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 8
  30. 2022Contextual Transformer for Offline Meta Reinforcement Learningcited by 7
  31. 2024CoSLight: Co-optimizing Collaborator Selection and Decision-making to Enhance Traffic Signal Controlcited by 7
  32. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
  33. 2025Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learningcited by 7
  34. 2026Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agentscited by 7
  35. 2023Scalable Communication for Multi-Agent Reinforcement Learning via Transformer-Based Email Mechanismcited by 6
  36. 2023Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigationcited by 6
  37. 2023MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learningcited by 6
  38. 2024Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaborationcited by 6

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning
  2. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  3. 2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models
  4. 2025Transformer-Based Multi-Agent Reinforcement Learning for Networked Systems with Long-Range Interactions
  5. 2024MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Search
  6. 2024Generalising Multi-Agent Cooperation through Task-Agnostic Communication
  7. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policy
  8. 2026Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning