mlsanityData to 7 Oct 2026

Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training

arXiv 2412.12496 ↗First posted 17 Dec 2024cs.CV cs.AIcited by 5

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

also in 1 more
Abstract (arXiv)

Vision Mamba has shown close to state of the art performance on computer vision tasks, drawing much interest in increasing it's efficiency. A promising approach is token reduction (that has been successfully implemented in ViTs). Pruning informative tokens in Mamba leads to a high loss of key knowledge and degraded performance. An alternative, of merging tokens preserves more information than pruning, also suffers for large compression ratios. Our key insight is that a quick round of retraining after token merging yeilds robust results across various compression ratios. Empirically, pruned Vims only drop up to 0.9% accuracy on ImageNet-1K, recovered by our proposed framework R-MeeTo in our main evaluation. We show how simple and effective the fast recovery can be achieved at minute-level, in particular, a 35.9% accuracy spike over 3 epochs of training on Vim-Ti. Moreover, Vim-Ti/S/B are re-trained within 5/7/17 minutes, and Vim-S only drops 1.3% with 1.2x (up to 1.5x) speed up in inference.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 46 of its 54 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

46 of 54 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbone10 Jul
  2. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition26 Mar
  3. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series22 Mar
  4. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning20 Mar
  5. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba15 Mar
  6. 2024LocalMamba: Visual State Space Model with Windowed Selective Scan14 Mar
  7. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model17 Jan

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  7. 2022Token Merging: Your ViT But Fastercited by 995
  8. 2024VMamba: Visual State Space Modelcited by 955
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  12. 2018MINE: Mutual Information Neural Estimationcited by 680
27 more
  1. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  2. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  3. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  4. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  5. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  6. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  7. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  8. 2022Long Range Language Modeling via Gated State Spacescited by 265
  9. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  10. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  11. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  12. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  13. 2021Learned Token Pruning for Transformerscited by 138
  14. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  15. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  16. 2024RSMamba: Remote Sensing Image Classification with State Space Modelcited by 95
  17. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
  18. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  19. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  20. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  21. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  22. 2021Token Pooling in Vision Transformerscited by 68
  23. 2024Fusion-Mamba for Cross-modality Object Detectioncited by 67
  24. 2024Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Modelcited by 62
  25. 2024Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Modelcited by 57
  26. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  27. 2024Exploring Token Pruning in Vision State Space Modelscited by 27

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processingcited by 0
  2. 2025Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learningcited by 0
  3. 2025QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Modelscited by 0
  4. 2025Sequential Token Merging: Revisiting Hidden Statescited by 0
  5. 2025MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mambacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Vision Mamba
  2. 2025Training-free Token Reduction for Vision Mamba
  3. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion
  4. 2026MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis
  5. 2025Post-Training Quantization for Vision Mamba with k-Scaled Quantization and Reparameterization
  6. 2024PTQ4VM: Post-Training Quantization for Visual Mamba
  7. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient
  8. 2025Lossless Token Merging Even Without Fine-Tuning in Vision Transformers