# Large Sequence Models for Sequential Decision-Making: A Survey

arXiv: https://arxiv.org/abs/2306.13945 · HTML: https://mlsanity.com/paper/2306.13945/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 89 of its 129 references that are in the map.
- First posted on arXiv 24 Jun 2023.

## Builds on (most cited first, 30 of 89)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)

## Built on by (most cited first, 19 of 19)

- [MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control](https://mlsanity.com/paper/2403.12037/) (2024)
- [CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2401.17043/) (2024)
- [Critic-Guided Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2312.13716/) (2023)
- [ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning](https://mlsanity.com/paper/2309.05915/) (2023)
- [The Duality of Generative AI and Reinforcement Learning in Robotics: A Review](https://mlsanity.com/paper/2410.16411/) (2024)
- [Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models](https://mlsanity.com/paper/2407.03964/) (2024)
- [Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Models](https://mlsanity.com/paper/2411.14457/) (2024)
- [Adaptive Control Strategy for Quadruped Robots in Actuator Degradation Scenarios](https://mlsanity.com/paper/2312.17606/) (2023)
- [Self-supervised Pretraining for Decision Foundation Model: Formulation, Pipeline and Challenges](https://mlsanity.com/paper/2401.00031/) (2023)
- [Machine Learning for Blockchain Data Analysis: Progress and Opportunities](https://mlsanity.com/paper/2404.18251/) (2024)
- [Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation](https://mlsanity.com/paper/2607.06564/) (2026)
- [Inverse Design of Photonic Crystal Surface Emitting Lasers is a Sequence Modeling Problem](https://mlsanity.com/paper/2403.05149/) (2024)
- [Task-agnostic Decision Transformer for Multi-type Agent Control with Federated Split Training](https://mlsanity.com/paper/2405.13445/) (2024)
- [PMAT: Optimizing Action Generation Order in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.16496/) (2025)
- [Decision SpikeFormer: Spike-Driven Transformer for Decision Making](https://mlsanity.com/paper/2504.03800/) (2025)
- [From Sorting Algorithms to Scalable Kernels: Bayesian Optimization in High-Dimensional Permutation Spaces](https://mlsanity.com/paper/2507.13263/) (2025)
- [A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits](https://mlsanity.com/paper/2601.12945/) (2026)
- [CroSTAta: Cross-State Transition Attention Transformer for Robotic Manipulation](https://mlsanity.com/paper/2510.00726/) (2025)
- [Exploring In-Context Learning for Handwritten Text Recognition](https://mlsanity.com/paper/2609.37195/) (2026)

## Abstract (arXiv)

Transformer architectures have facilitated the development of large-scale and general-purpose sequence models for prediction tasks in natural language processing and computer vision, e.g., GPT-3 and Swin Transformer. Although originally designed for prediction problems, it is natural to inquire about their suitability for sequential decision-making and reinforcement learning problems, which are typically beset by long-standing issues involving sample efficiency, credit assignment, and partial observability. In recent years, sequence models, especially the Transformer, have attracted increasing interest in the RL communities, spawning numerous approaches with notable effectiveness and generalizability. This survey presents a comprehensive overview of recent works aimed at solving sequential decision-making tasks with sequence models such as the Transformer, by discussing the connection between sequential decision-making and sequence modeling, and categorizing them based on the way they utilize the Transformer. Moreover, this paper puts forth various potential avenues for future research intending to improve the effectiveness of large sequence models for sequential decision-making, encompassing theoretical foundations, network architectures, algorithms, and efficient training systems. As this article has been accepted by the Frontiers of Computer Science, here is an early version, and the most up-to-date version can be found at https://journal.hep.com.cn/fcs/EN/10.1007/s11704-023-2689-5