mlsanityData to 14 Oct 2026

Sequential Token Merging: Revisiting Hidden States

arXiv 2509.22691 ↗First posted 19 Sep 2025cs.CVcited by 0

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

also in 1 more
Abstract (arXiv)

Vision Mambas (ViMs) achieve remarkable success with sub-quadratic complexity, but their efficiency remains constrained by quadratic token scaling with image resolution. While existing methods address token redundancy, they overlook ViMs' intrinsic Limited Directional Sequential Dependence (LDSD) - a critical information flow mechanism revealed in our analysis. We further identify Mamba's selective scan enables gradual information aggregation in hidden states. Based on these insights, we propose Sequential Token Merging (STM), featuring: 1) Bidirectional nearest neighbor merging to preserve sequential dependencies through symmetric spatial aggregation, and 2) Hidden states protection to stabilize the hidden states around the class token. STM strategically leverages Mamba's layer-wise loss convergence to convert temporal forgetfulness into stability. Experiments demonstrate STM's superiority: 1.0% accuracy drop for ViM-Ti at 20% token reduction, and only 1.4% degradation for ViM-S at 40% reduction. Our method achieves state-of-the-art efficiency with minimal complexity, while providing new insights into state-space model dynamics. Codes will be released soon.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 43 of its 45 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

43 of 45 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025OuroMamba: A Data-Free Quantization Framework for Vision Mamba13 Mar
  2. 2025Post-Training Quantization for Vision Mamba with k-Scaled Quantization and Reparameterization28 Jan
  3. 2024PTQ4VM: Post-Training Quantization for Visual Mamba29 Dec
  4. 2024Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training17 Dec
  5. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion15 Sep
  6. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbone10 Jul
  7. 2024Demystify Mamba in Vision: A Linear Attention Perspective26 May
  8. 2024Vision Mamba: A Comprehensive Survey and Taxonomy7 May
  9. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition26 Mar
  10. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series22 Mar
  11. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba15 Mar
  12. 2024LocalMamba: Visual State Space Model with Windowed Selective Scan14 Mar
1 more
  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model17 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  10. 2022Token Merging: Your ViT But Fastercited by 1,021
  11. 2024VMamba: Visual State Space Modelcited by 969
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
18 more
  1. 2020A Survey on Visual Transformercited by 654
  2. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  3. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  4. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  5. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  6. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 287
  7. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  8. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  9. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  10. 2024RSMamba: Remote Sensing Image Classification with State Space Modelcited by 95
  11. 2024The Hidden Attention of Mamba Modelscited by 91
  12. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 81
  13. 2022Learning to Merge Tokens in Vision Transformerscited by 65
  14. 2024Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Modelcited by 62
  15. 2024RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentationcited by 33
  16. 2024Exploring Token Pruning in Vision State Space Modelscited by 28
  17. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 28
  18. 2024Rethinking Token Reduction for State Space Modelscited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Vision State Space Models via Token Merging
  2. 2026LapMerge: Differentiable Exact-Budget Token Merging in Linear Time
  3. 2025CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones
  4. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
  5. 2024Demystifying the Token Dynamics of Deep Selective State Space Models
  6. 2024Attamba: Attending To Multi-Token States
  7. 2026Spatial-Aware Reduction Framework: Towards Efficient and Faithful Visual State Space Models
  8. 2026Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice