# Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling arXiv: https://arxiv.org/abs/2406.00079 · HTML: https://mlsanity.com/paper/2406.00079/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). - Builds on 37 of its 47 references that are in the map. - First posted on arXiv 31 May 2024. ## Placed in - State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/) - Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/) ## Builds on (most cited first, 30 of 37) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023) - [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022) - [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013) - [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021) - [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020) - [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021) - [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020) - [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018) - [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024) - [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018) - [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021) - [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019) - [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022) - [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022) - [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021) - [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016) - [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022) - [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018) - [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016) - [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021) - [Long Range Language Modeling via Gated State Spaces](https://mlsanity.com/paper/2206.13947/) (2022) - [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022) - [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023) - [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020) ## Built on by (most cited first, 18 of 18) - [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025) - [Overflow Prevention Enhances Long-Context Recurrent LLMs](https://mlsanity.com/paper/2505.07793/) (2025) - [In-Context Reinforcement Learning via Communicative World Models](https://mlsanity.com/paper/2508.06659/) (2025) - [Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient](https://mlsanity.com/paper/2410.08893/) (2024) - [EDELINE: Enhancing Memory in Diffusion-based World Models via Linear-Time Sequence Modeling](https://mlsanity.com/paper/2502.00466/) (2025) - [Towards Provable Emergence of In-Context Reinforcement Learning](https://mlsanity.com/paper/2509.18389/) (2025) - [Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2505.01822/) (2025) - [Generative AI for Intent-Driven Network Management in 6G RAN: A Case Study on the Mamba Model](https://mlsanity.com/paper/2508.06616/) (2025) - [X-VMamba: Explainable Vision Mamba](https://mlsanity.com/paper/2511.12694/) (2025) - [Decoupling Return-to-Go for Efficient Decision Transformer](https://mlsanity.com/paper/2601.15953/) (2026) - [DADP: Domain Adaptive Diffusion Policy](https://mlsanity.com/paper/2602.04037/) (2026) - [Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation](https://mlsanity.com/paper/2603.15134/) (2026) - [Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing](https://mlsanity.com/paper/2408.10517/) (2024) - [An Uncertainty-Weighted Decision Transformer for Navigation in Dense, Complex Driving Scenarios](https://mlsanity.com/paper/2509.13132/) (2025) - [Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing](https://mlsanity.com/paper/2602.19805/) (2026) - [Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control](https://mlsanity.com/paper/2603.03932/) (2026) - [Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer](https://mlsanity.com/paper/2605.06104/) (2026) - [MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddings](https://mlsanity.com/paper/2605.17431/) (2026) ## Abstract (arXiv) Recent works have shown the remarkable superiority of transformer models in reinforcement learning (RL), where the decision-making problem is formulated as sequential generation. Transformer-based agents could emerge with self-improvement in online environments by providing task contexts, such as multiple trajectories, called in-context RL. However, due to the quadratic computation complexity of attention in transformers, current in-context RL methods suffer from huge computational costs as the task horizon increases. In contrast, the Mamba model is renowned for its efficient ability to process long-term dependencies, which provides an opportunity for in-context RL to solve tasks that require long-term memory. To this end, we first implement Decision Mamba (DM) by replacing the backbone of Decision Transformer (DT). Then, we propose a Decision Mamba-Hybrid (DM-H) with the merits of transformers and Mamba in high-quality prediction and long-term memory. Specifically, DM-H first generates high-value sub-goals from long-term memory through the Mamba model. Then, we use sub-goals to prompt the transformer, establishing high-quality predictions. Experimental results demonstrate that DM-H achieves state-of-the-art in long and short-term tasks, such as D4RL, Grid World, and Tmaze benchmarks. Regarding efficiency, the online testing of DM-H in the long-term task is 28× times faster than the transformer-based baselines.