# Rethinking Token Reduction for State Space Models

arXiv: https://arxiv.org/abs/2410.14725 · HTML: https://mlsanity.com/paper/2410.14725/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)
- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Long Range Language Modeling via Gated State Spaces](https://mlsanity.com/paper/2206.13947/) (2022)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [Learning to Merge Tokens in Vision Transformers](https://mlsanity.com/paper/2202.12015/) (2022)
- [PuMer: Pruning and Merging Tokens for Efficient Vision Language Models](https://mlsanity.com/paper/2305.17530/) (2023)
- [Exploring Token Pruning in Vision State Space Models](https://mlsanity.com/paper/2409.18962/) (2024)
- [AdapLeR: Speeding up Inference by Adaptive Length Reduction](https://mlsanity.com/paper/2203.08991/) (2022)
- [Squat: Quant Small Language Models on the Edge](https://mlsanity.com/paper/2402.10787/) (2024)
- [Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection](https://mlsanity.com/paper/2203.14380/) (2022)
- [Optimizing Retrieval-augmented Reader Models via Token Elimination](https://mlsanity.com/paper/2310.13682/) (2023)

## Built on by (most cited first, 15 of 15)

- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting](https://mlsanity.com/paper/2507.05116/) (2025)
- [Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering](https://mlsanity.com/paper/2505.10118/) (2025)
- [RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation](https://mlsanity.com/paper/2501.04315/) (2025)
- [QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edge](https://mlsanity.com/paper/2503.16709/) (2025)
- [Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLA](https://mlsanity.com/paper/2512.22208/) (2025)
- [Collaborative Compression for Large-Scale MoE Deployment on Edge](https://mlsanity.com/paper/2509.25689/) (2025)
- [Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation](https://mlsanity.com/paper/2505.23844/) (2025)
- [PerfMamba: Performance Analysis and Pruning of Selective State Space Models](https://mlsanity.com/paper/2511.22849/) (2025)
- [Vision SmolMamba: Spike-Guided Token Pruning for Energy-Efficient Spiking State-Space Vision Models](https://mlsanity.com/paper/2604.25570/) (2026)
- [Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing](https://mlsanity.com/paper/2502.00594/) (2025)
- [Sequential Token Merging: Revisiting Hidden States](https://mlsanity.com/paper/2509.22691/) (2025)
- [Fine-Tuning Qwen3-27B for C-to-Rust Code Translation: A Three-Stage Curriculum of Pretraining, Debugging-Aware SFT, and Task-Specific SFT](https://mlsanity.com/paper/2608.13681/) (2026)
- [LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception](https://mlsanity.com/paper/2609.39938/) (2026)
- [From Patching to Pruning Visual Computation in Vision Language Models](https://mlsanity.com/paper/2610.03389/) (2026)

## Abstract (arXiv)

Recent advancements in State Space Models (SSMs) have attracted significant interest, particularly in models optimized for parallel training and handling long-range dependencies. Architectures like Mamba have scaled to billions of parameters with selective SSM. To facilitate broader applications using Mamba, exploring its efficiency is crucial. While token reduction techniques offer a straightforward post-training strategy, we find that applying existing methods directly to SSMs leads to substantial performance drops. Through insightful analysis, we identify the reasons for this failure and the limitations of current techniques. In response, we propose a tailored, unified post-training token reduction method for SSMs. Our approach integrates token importance and similarity, thus taking advantage of both pruning and merging, to devise a fine-grained intra-layer token reduction strategy. Extensive experiments show that our method improves the average accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods, while significantly reducing computational demands and memory requirements.