# Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges

arXiv: https://arxiv.org/abs/2404.16112 · HTML: https://mlsanity.com/paper/2404.16112/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 165 of its 192 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 165)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)

## Built on by (most cited first, 30 of 64)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Retinexmamba: Retinex-based Mamba for Low-light Image Enhancement](https://mlsanity.com/paper/2405.03349/) (2024)
- [Vision Mamba: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2405.04404/) (2024)
- [Were RNNs All We Needed?](https://mlsanity.com/paper/2410.01201/) (2024)
- [Matten: Video Generation with Mamba-Attention](https://mlsanity.com/paper/2405.03025/) (2024)
- [Rethinking Scanning Strategies with Vision Mamba in Semantic Segmentation of Remote Sensing Imagery: An Experimental Study](https://mlsanity.com/paper/2405.08493/) (2024)
- [MVGamba: Unify 3D Content Generation as State Space Sequence Modeling](https://mlsanity.com/paper/2406.06367/) (2024)
- [Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02573/) (2024)
- [xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories](https://mlsanity.com/paper/2410.16928/) (2024)
- [Long-Form Speech Generation with Spoken Language Models](https://mlsanity.com/paper/2412.18603/) (2024)
- [SMamba: Sparse Mamba for Event-based Object Detection](https://mlsanity.com/paper/2501.11971/) (2025)
- [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024)
- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [Comba: Improving Bilinear RNNs with Closed-loop Control](https://mlsanity.com/paper/2506.02475/) (2025)
- [MambaDS: Near-Surface Meteorological Field Downscaling with Topography Constrained Selective State Space Modeling](https://mlsanity.com/paper/2408.10854/) (2024)
- [Snakes and Ladders: Two Steps Up for VideoMamba](https://mlsanity.com/paper/2406.19006/) (2024)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba](https://mlsanity.com/paper/2503.18970/) (2025)
- [Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models](https://mlsanity.com/paper/2406.04320/) (2024)
- [A Survey on Mamba Architecture for Vision Applications](https://mlsanity.com/paper/2502.07161/) (2025)
- [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025)
- [Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics](https://mlsanity.com/paper/2409.08530/) (2024)
- [Benchmarking ECG FMs: A Reality Check Across Clinical Tasks](https://mlsanity.com/paper/2509.25095/) (2025)
- [RotRNN: Modelling Long Sequences with Rotations](https://mlsanity.com/paper/2407.07239/) (2024)
- [A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing](https://mlsanity.com/paper/2409.15022/) (2024)
- [Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook](https://mlsanity.com/paper/2505.00630/) (2025)
- [JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model](https://mlsanity.com/paper/2505.17257/) (2025)
- [Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs](https://mlsanity.com/paper/2506.07919/) (2025)
- [Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention](https://mlsanity.com/paper/2507.00449/) (2025)
- [CAMS: Convolution and Attention-Free Mamba-based Cardiac Image Segmentation](https://mlsanity.com/paper/2406.05786/) (2024)

## Abstract (arXiv)

Sequence modeling is a crucial area across various domains, including Natural Language Processing (NLP), speech recognition, time series forecasting, music generation, and bioinformatics. Recurrent Neural Networks (RNNs) and Long Short Term Memory Networks (LSTMs) have historically dominated sequence modeling tasks like Machine Translation, Named Entity Recognition (NER), etc. However, the advancement of transformers has led to a shift in this paradigm, given their superior performance. Yet, transformers suffer from O(N2) attention complexity and challenges in handling inductive bias. Several variations have been proposed to address these issues which use spectral networks or convolutions and have performed well on a range of tasks. However, they still have difficulty in dealing with long sequences. State Space Models(SSMs) have emerged as promising alternatives for sequence modeling paradigms in this context, especially with the advent of S4 and its variants, such as S4nd, Hippo, Hyena, Diagnol State Spaces (DSS), Gated State Spaces (GSS), Linear Recurrent Unit (LRU), Liquid-S4, Mamba, etc. In this survey, we categorize the foundational SSMs based on three paradigms namely, Gating architectures, Structural architectures, and Recurrent architectures. This survey also highlights diverse applications of SSMs across domains such as vision, video, audio, speech, language (especially long sequence modeling), medical (including genomics), chemical (like drug design), recommendation systems, and time series analysis, including tabular data. Moreover, we consolidate the performance of SSMs on benchmark datasets like Long Range Arena (LRA), WikiText, Glue, Pile, ImageNet, Kinetics-400, sstv2, as well as video datasets such as Breakfast, COIN, LVU, and various time series datasets. The project page for Mamba-360 work is available on this webpage.urlhttps://github.com/badripatro/mamba360.