# MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection

arXiv: https://arxiv.org/abs/2403.19888 · HTML: https://mlsanity.com/paper/2403.19888/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 111 of its 140 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 111)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)

## Built on by (most cited first, 30 of 40)

- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [Nested Learning: The Illusion of Deep Learning Architectures](https://mlsanity.com/paper/2512.24695/) (2025)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization](https://mlsanity.com/paper/2504.13173/) (2025)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [Mamba-R: Vision Mamba ALSO Needs Registers](https://mlsanity.com/paper/2405.14858/) (2024)
- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining](https://mlsanity.com/paper/2405.19450/) (2024)
- [CU-Mamba: Selective State Space Models with Channel Learning for Image Restoration](https://mlsanity.com/paper/2404.11778/) (2024)
- [VFIMamba: Video Frame Interpolation with State Space Models](https://mlsanity.com/paper/2407.02315/) (2024)
- [Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports](https://mlsanity.com/paper/2405.17680/) (2024)
- [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024)
- [QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model](https://mlsanity.com/paper/2410.06806/) (2024)
- [Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models](https://mlsanity.com/paper/2410.09385/) (2024)
- [Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting](https://mlsanity.com/paper/2405.16312/) (2024)
- [Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models](https://mlsanity.com/paper/2406.04320/) (2024)
- [AC-MAMBASEG: An adaptive convolution and Mamba-based architecture for enhanced skin lesion segmentation](https://mlsanity.com/paper/2405.03011/) (2024)
- [Best of Both Worlds: Advantages of Hybrid Graph Sequence Models](https://mlsanity.com/paper/2411.15671/) (2024)
- [Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook](https://mlsanity.com/paper/2505.00630/) (2025)
- [Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers](https://mlsanity.com/paper/2512.03870/) (2025)
- [Alignment-free Raw Video Demoireing](https://mlsanity.com/paper/2408.10679/) (2024)
- [Mamba-CAD: State Space Model For 3D Computer-Aided Design Generative Modeling](https://mlsanity.com/paper/2603.00439/) (2026)
- [HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet](https://mlsanity.com/paper/2604.14724/) (2026)
- [Enhanced Prediction of Multi-Agent Trajectories via Control Inference and State-Space Dynamics](https://mlsanity.com/paper/2408.12609/) (2024)
- [EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment](https://mlsanity.com/paper/2410.05938/) (2024)
- [Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models](https://mlsanity.com/paper/2503.11224/) (2025)
- [MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling](https://mlsanity.com/paper/2512.23824/) (2025)
- [Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing](https://mlsanity.com/paper/2601.00245/) (2026)

## Abstract (arXiv)

Recent advances in deep learning have mainly relied on Transformers due to their data dependency and ability to learn at scale. The attention module in these architectures, however, exhibits quadratic time and space in input size, limiting their scalability for long-sequence modeling. Despite recent attempts to design efficient and effective architecture backbone for multi-dimensional data, such as images and multivariate time series, existing models are either data independent, or fail to allow inter- and intra-dimension communication. Recently, State Space Models (SSMs), and more specifically Selective State Space Models, with efficient hardware-aware implementation, have shown promising potential for long sequence modeling. Motivated by the success of SSMs, we present MambaMixer, a new architecture with data-dependent weights that uses a dual selection mechanism across tokens and channels, called Selective Token and Channel Mixer. MambaMixer connects selective mixers using a weighted averaging mechanism, allowing layers to have direct access to early features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2) architectures based on the MambaMixer block and explore their performance in various vision and time series forecasting tasks. Our results underline the importance of selective mixing across both tokens and channels. In ImageNet classification, object detection, and semantic segmentation tasks, ViM2 achieves competitive performance with well-established vision models and outperforms SSM-based vision models. In time series forecasting, TSM2 achieves outstanding performance compared to state-of-the-art methods while demonstrating significantly improved computational cost. These results show that while Transformers, cross-channel attention, and MLPs are sufficient for good performance in time series forecasting, neither is necessary.