mlsanityData to 7 Oct 2026

Multi-Game Decision Transformers

arXiv 2205.15241 ↗First posted 30 May 2022cs.AI cs.LGcited by 212

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

A longstanding goal of the field of AI is a method for learning a highly capable, generalist agent from diverse experience. In the subfields of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets. Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents. Specifically, we show that a single transformer-based model - with a single set of weights - trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human performance. When trained and evaluated appropriately, we find that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via fine-tuning. We compare several approaches in this multi-game setting, such as online and offline RL methods and behavioral cloning, and find that our Multi-Game Decision Transformer models offer the best scalability and performance. We release the pre-trained models and code to encourage further research in this direction.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 69 of its 98 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

69 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  2. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  13. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  14. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  15. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  16. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  17. 2020Mastering Atari with Discrete World Modelscited by 990
  18. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  19. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  20. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  21. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  22. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  23. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  24. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  25. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  26. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  27. 2015Policy Distillationcited by 414
  28. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  29. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  30. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  31. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  32. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  33. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  34. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  35. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  36. 2021Embodied Intelligence via Learning and Evolutioncited by 154
  37. 2021Discovering and Achieving Goals via World Modelscited by 131
  38. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 71; 2024: 60; 2025: 36; 2026: 23.

212 papers.

Sort:

By area:

  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  5. 2023Towards Generalist Biomedical AIcited by 286
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  7. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  8. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  9. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  10. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  11. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  12. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
38 more
  1. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  2. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  3. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  4. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  5. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  6. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  7. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  8. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  9. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  10. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  11. 2023Elastic Decision Transformercited by 66
  12. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  13. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  14. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  15. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  16. 2025Horizon Reduction Makes RL Scalablecited by 55
  17. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  18. 2023Anti-Exploration by Random Network Distillationcited by 49
  19. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  20. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  21. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  22. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  23. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  24. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
  25. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  26. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  27. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  28. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  29. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  30. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  31. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  32. 2023Barkour: Benchmarking Animal-level Agility with Quadruped Robotscited by 33
  33. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  34. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 32
  35. 2022Large Language Models can Implement Policy Iterationcited by 31
  36. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  37. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  38. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games
  2. 2024Learning to Play Atari in a World of Tokens
  3. 2025Do We Need Transformers to Play FPS Video Games?
  4. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  5. 2025Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning
  6. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
  7. 2024Large Language Models as Agents in Two-Player Games
  8. 2022Transformers are Sample-Efficient World Models