mlsanityData to 7 Oct 2026

Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models

arXiv 2501.17088 ↗First posted 28 Jan 2025cs.LG cs.AI cs.CLcited by 7

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Large pre-trained models have achieved outstanding results in sequence modeling. The Transformer block and its attention mechanism have been the main drivers of the success of these models. Recently, alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address the inefficiencies of Transformers. This paper explores the compression of SSM-based models, particularly Mamba and its hybrids. We study the sensitivity of these models to the removal of selected components at different granularities to reduce the model size and computational overhead, thus improving their efficiency while maintaining accuracy. The proposed solutions, collectively referred to as Mamba-Shedder, achieve a speedup of up to 1.4x during inference, demonstrating that model efficiency can be improved by eliminating several redundancies with minimal impact on the overall model performance. The code is available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 34 of its 42 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

34 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Model7 Oct
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
19 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  6. 2018Image Transformercited by 770
  7. 2021AST: Audio Spectrogram Transformercited by 634
  8. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  11. 2021Block Pruning For Faster Transformerscited by 176
  12. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  13. 2019Adaptively Sparse Transformerscited by 154
  14. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  15. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 44
  16. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 35
  17. 2022Vision Transformers: From Semantic Segmentation to Dense Predictioncited by 6
  18. 2024Shears: Unstructured Sparsity with Neural Low-rank Adapter Searchcited by 4
  19. 2025MultiPruner: Balanced Structure Removal in Foundation Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  2. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  3. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  4. 2025Hankel Singular Value Regularization for Highly Compressible State Space Modelscited by 2
  5. 2025PerfMamba: Performance Analysis and Pruning of Selective State Space Modelscited by 1
  6. 2026S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devicescited by 0
  7. 2026What Does Layer-Importance Reveal About Transformers and State-Space Models?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Mamba
  2. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  3. 2024An Empirical Study of Mamba-based Language Models
  4. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data
  5. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
  6. 2026Mamba-3: Improved Sequence Modeling using State Space Principles
  7. 2024Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models
  8. 2025Block-Biased Mamba for Long-Range Sequence Processing