mlsanityData to 14 Oct 2026

On Pruning State-Space LLMs

arXiv 2502.18886 ↗First posted 26 Feb 2025cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g. WANDA), while using other methods lead to fast performance degradation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

20 of 26 references.

References · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
8 more
  1. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  2. 2019Importance Estimation for Neural Network Pruningcited by 541
  3. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  4. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  6. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  7. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  8. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQAcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  2. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  3. 2025Hankel Singular Value Regularization for Highly Compressible State Space Modelscited by 2
  4. 2025Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruningcited by 1
  5. 2026GHOST: Unmasking Phantom States in Mamba2 via Grouped Hidden-state Output-aware Selection & Truncationcited by 1
  6. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardwarecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
  2. 2025PerfMamba: Performance Analysis and Pruning of Selective State Space Models
  3. 2026AIRE-Prune: Asymptotic Impulse-Response Energy for State Pruning in State Space Models
  4. 2024Layer-Adaptive State Pruning for Deep State Space Models
  5. 2024Exploring Token Pruning in Vision State Space Models
  6. 2025State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
  7. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  8. 2024State Space Model for New-Generation Network Alternative to Transformers: A Survey