mlsanityData to 14 Oct 2026

StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning

arXiv 2110.06206 ↗First posted 12 Oct 2021cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) can be considered as a sequence modeling task: given a sequence of past state-action-reward experiences, an agent predicts a sequence of next actions. In this work, we propose State-Action-Reward Transformer (StARformer) for visual RL, which explicitly models short-term state-action-reward representations (StAR-representations), essentially introducing a Markovian-like inductive bias to improve long-term modeling. Our approach first extracts StAR-representations by self-attending image state patches, action, and reward tokens within a short temporal window. These are then combined with pure image state representations -- extracted as convolutional features, to perform self-attention over the whole sequence. Our experiments show that StARformer outperforms the state-of-the-art Transformer-based method on image-based Atari and DeepMind Control Suite benchmarks, in both offline-RL and imitation learning settings. StARformer is also more compliant with longer sequences of inputs. Our code is available at https://github.com/elicassion/StARformer.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

46 of 69 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2020Reformer: The Efficient Transformercited by 1,668
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2020Rethinking Attention with Performerscited by 1,601
34 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,597
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2021Video Swin Transformercited by 1,091
  6. 2020Mastering Atari with Discrete World Modelscited by 1,007
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2021Going deeper with Image Transformerscited by 764
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2021Transformer in Transformercited by 633
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  14. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  15. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  16. 2020Reinforcement Learning with Augmented Datacited by 468
  17. 2021Early Convolutions Help Transformers See Bettercited by 396
  18. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  19. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  20. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  21. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  22. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  23. 2021Video Transformer Networkcited by 272
  24. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  25. 2018Generative Adversarial Imitation from Observationcited by 184
  26. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  27. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  28. 2019Reward-Conditioned Policiescited by 102
  29. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  30. 2020Transformers for One-Shot Visual Imitationcited by 78
  31. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 60
  32. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  33. 2021The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learningcited by 50
  34. 2021Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learningcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 11; 2025: 4; 2026: 2.

27 papers.

Sort:

By area:

  1. 2024Language Repository for Long Video Understandingcited by 55
  2. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  3. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  4. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 40
  5. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  6. 2023Learning to Modulate pre-trained Models in RLcited by 28
  7. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  8. 2023Graph Decision Transformercited by 20
  9. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  10. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  11. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  12. 2023PLEX: Making the Most of the Available Data for Robotic Manipulation Pretrainingcited by 9
15 more
  1. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  2. 2023PDiT: Interleaving Perception and Decision-making Transformers for Deep Reinforcement Learningcited by 5
  3. 2024Return-Aligned Decision Transformercited by 3
  4. 2024Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Headscited by 3
  5. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
  6. 2024MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spacescited by 1
  7. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Makingcited by 1
  8. 2023Active Vision Reinforcement Learning under Limited Visual Observabilitycited by 0
  9. 2024Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spacescited by 0
  10. 2024Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  11. 2025Target Return Optimizer for Multi-Game Decision Transformercited by 0
  12. 2025Playing Non-Embedded Card-Based Games with Reinforcement Learningcited by 0
  13. 2025SAINT: Attention-Based Policies for Discrete Combinatorial Action Spacescited by 0
  14. 2026Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  15. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learning
  2. 2024Pretrained Visual Representations in Reinforcement Learning
  3. 2025Reinforcement Learning for Large Model: A Survey
  4. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
  5. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
  6. 2025A Survey of State Representation Learning for Deep Reinforcement Learning
  7. 2026Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
  8. 2024Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning