mlsanityData to 7 Oct 2026

Decision S4: Efficient Sequence-Based RL via State Spaces Layers

arXiv 2306.05167 ↗First posted 8 Jun 2023cs.LGcited by 26

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recently, sequence learning methods have been applied to the problem of off-policy Reinforcement Learning, including the seminal work on Decision Transformers, which employs transformers for this task. Since transformers are parameter-heavy, cannot benefit from history longer than a fixed window size, and are not computed using recurrence, we set out to investigate the suitability of the S4 family of models, which are based on state-space layers and have been shown to outperform transformers, especially in modeling long-range dependencies. In this work we present two main algorithms: (i) an off-policy training procedure that works with trajectories, while still maintaining the training efficiency of the S4 model. (ii) An on-policy training procedure that is trained in a recurrent manner, benefits from long-range dependencies, and is based on a novel stable actor-critic mechanism. Our results indicate that our method outperforms multiple variants of decision transformers, as well as the other baseline methods on most tasks, while reducing the latency, number of parameters, and training time by several orders of magnitude, making our approach more suitable for real-world RL.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 68 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

38 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Multi-Game Decision Transformers30 May
  2. 2022Online Decision Transformer11 Feb
  3. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2022A Generalist Agentcited by 802
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
20 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  3. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  4. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  6. 2022Long Range Language Modeling via Gated State Spacescited by 265
  7. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  8. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  9. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  10. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  11. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  12. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  13. 2021Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learningcited by 73
  14. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  15. 2020Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?cited by 34
  16. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  17. 2015Recurrent Reinforcement Learning: A Hybrid Approachcited by 13
  18. 2020Flexible and Efficient Long-Range Planning Through Curious Explorationcited by 4
  19. 2019Variance Reduction in Actor Critic Methods (ACM)cited by 1
  20. 2020Long-Term Planning with Deep Reinforcement Learning on Autonomous Dronescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 16; 2025: 2; 2026: 2.

26 papers.

Sort:

By area:

  1. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  2. 2024The Hidden Attention of Mamba Modelscited by 88
  3. 2024Mastering Memory Tasks with World Modelscited by 38
  4. 2024Were RNNs All We Needed?cited by 36
  5. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  6. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  7. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  8. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  9. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  10. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spacescited by 23
  11. 2023Recurrent Action Transformer with Memorycited by 17
  12. 2024Slot State Space Modelscited by 17
14 more
  1. 2024SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spacescited by 13
  2. 2023On the Long Range Abilities of Transformerscited by 10
  3. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  4. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  5. 2023Multi-Dimensional Hyena for Spatial Inductive Biascited by 5
  6. 2024Return-Aligned Decision Transformercited by 3
  7. 2026Fatigue-Aware Learning to Defer via Constrained Optimisationcited by 3
  8. 2024Context-aware Mamba-based Reinforcement Learning for social robot navigationcited by 2
  9. 2025Differential Mambacited by 2
  10. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learningcited by 1
  11. 2024Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  12. 2024Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Gamescited by 0
  13. 2025Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learningcited by 0
  14. 2026Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Structured State Space Models for In-Context Reinforcement Learning
  2. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
  3. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  4. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  5. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  7. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
  8. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning