# Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces

arXiv: https://arxiv.org/abs/2403.19925 · HTML: https://mlsanity.com/paper/2403.19925/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)
- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 14 of 14)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Decision S4: Efficient Sequence-Based RL via State Spaces Layers](https://mlsanity.com/paper/2306.05167/) (2023)
- [Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making](https://mlsanity.com/paper/2310.03022/) (2023)

## Built on by (most cited first, 24 of 24)

- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Were RNNs All We Needed?](https://mlsanity.com/paper/2410.01201/) (2024)
- [TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning](https://mlsanity.com/paper/2410.09536/) (2024)
- [A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks](https://mlsanity.com/paper/2410.22391/) (2024)
- [Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?](https://mlsanity.com/paper/2405.12094/) (2024)
- [ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems](https://mlsanity.com/paper/2510.07151/) (2025)
- [Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient](https://mlsanity.com/paper/2410.08893/) (2024)
- [Demystifying the Token Dynamics of Deep Selective State Space Models](https://mlsanity.com/paper/2410.03292/) (2024)
- [Multi-Agent Reinforcement Learning with Selective State-Space Models](https://mlsanity.com/paper/2410.19382/) (2024)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [RLBenchNet: The Right Network for the Right Reinforcement Learning Task](https://mlsanity.com/paper/2505.15040/) (2025)
- [Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba](https://mlsanity.com/paper/2506.18184/) (2025)
- [Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning](https://mlsanity.com/paper/2406.02013/) (2024)
- [DADP: Domain Adaptive Diffusion Policy](https://mlsanity.com/paper/2602.04037/) (2026)
- [In-Context Reinforcement Learning under Non-Stationarity: A Survey](https://mlsanity.com/paper/2607.11906/) (2026)
- [Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing](https://mlsanity.com/paper/2408.10517/) (2024)
- [Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games](https://mlsanity.com/paper/2412.00725/) (2024)
- [Do We Need Transformers to Play FPS Video Games?](https://mlsanity.com/paper/2504.17891/) (2025)
- [Success in Humanoid Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2507.18883/) (2025)
- [Unified token representations for sequential decision models](https://mlsanity.com/paper/2510.21448/) (2025)
- [Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing](https://mlsanity.com/paper/2602.19805/) (2026)
- [Belief-State RWKV for Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2604.09671/) (2026)
- [QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL](https://mlsanity.com/paper/2605.01862/) (2026)

## Abstract (arXiv)

Decision Transformer, a promising approach that applies Transformer architectures to reinforcement learning, relies on causal self-attention to model sequences of states, actions, and rewards. While this method has shown competitive results, this paper investigates the integration of the Mamba framework, known for its advanced capabilities in efficient and effective sequence modeling, into the Decision Transformer architecture, focusing on the potential performance enhancements in sequential decision-making tasks. Our study systematically evaluates this integration by conducting a series of experiments across various decision-making environments, comparing the modified Decision Transformer, Decision Mamba, with its traditional counterpart. This work contributes to the advancement of sequential decision-making models, suggesting that the architecture and training methodology of neural networks can significantly impact their performance in complex tasks, and highlighting the potential of Mamba as a valuable tool for improving the efficacy of Transformer-based models in reinforcement learning scenarios.