mlsanityData to 14 Oct 2026

Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning

arXiv 2506.18537 ↗First posted 23 Jun 2025cs.LG cs.MAcited by 0

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

We present the Multi-Agent Transformer World Model (MATWM), a novel transformer-based world model designed for multi-agent reinforcement learning in both vector- and image-based environments. MATWM combines a decentralized imagination framework with a semi-centralized critic and a teammate prediction module, enabling agents to model and anticipate the behavior of others under partial observability. To address non-stationarity, we incorporate a prioritized replay mechanism that trains the world model on recent experiences, allowing it to adapt to agents' evolving policies. We evaluated MATWM on a broad suite of benchmarks, including the StarCraft Multi-Agent Challenge, PettingZoo, and MeltingPot. MATWM achieves state-of-the-art performance, outperforming both model-free and prior world model approaches, while demonstrating strong sample efficiency, achieving near-optimal performance in as few as 50K environment interactions. Ablation studies confirm the impact of each component, with substantial gains in coordination-heavy tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

38 of 47 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models22 Jun
  2. 2023Harnessing Discrete Representations For Continual Reinforcement Learning2 Dec
  3. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning14 Oct
  4. 2023Transformer-based World Models Are Happy With 100k Interactions13 Mar
  5. 2023Mastering Diverse Domains through World Models10 Jan
  6. 2022Transformers are Sample-Efficient World Models1 Sep
  7. 2022A Survey on Model-based Reinforcement Learning19 Jun
  8. 2022TransDreamer: Reinforcement Learning with Transformer World Models19 Feb
  9. 2021Mastering Atari Games with Limited Data30 Oct
  10. 2020Mastering Atari with Discrete World Models5 Oct
  11. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  12. 2018World Models27 Mar

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  10. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  11. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  12. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
14 more
  1. 2019The StarCraft Multi-Agent Challengecited by 597
  2. 2019Model-Based Reinforcement Learning for Ataricited by 502
  3. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  5. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  6. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  7. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  8. 2019Policy Consolidation for Continual Reinforcement Learningcited by 39
  9. 2022Scalable Multi-Agent Model-Based Reinforcement Learningcited by 25
  10. 2024Efficient Multi-agent Reinforcement Learning by Planningcited by 9
  11. 2023Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learningcited by 8
  12. 2022Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learningcited by 6
  13. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Taskscited by 5
  14. 2024Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025GAWM: Global-Aware World Model for Multi-Agent Reinforcement Learning
  2. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
  3. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Model
  4. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
  5. 2025Simulus: Combining Improvements in Sample-Efficient World Model Agents
  6. 2023MABL: Bi-Level Latent-Variable World Model for Sample-Efficient Multi-Agent Reinforcement Learning
  7. 2025From Pixels to Cooperation Multi Agent Reinforcement Learning based on Multimodal World Models
  8. 2022Multi-Game Decision Transformers