mlsanityData to 14 Oct 2026

Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals

arXiv 2604.16745 ↗First posted 17 Apr 2026cs.AI cs.CVcited by 0

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Training-free token reduction methods for Vision Transformers (ToMe, ToFu, PiToMe, and MCTF) employ different scoring mechanisms, yet they share a closely matched cliff-like collapse at high compression. This paper explains why. We develop a diagnostic framework with two tools, ranking consistency ρs and off-diagonal correlation ρoff, that decomposes the collapse into (1)a signal-agnostic error amplifier inherent to layer-wise reduction, predicting convex Pareto curves and rcrit propto 1/L; and (2)shared reliance on pairwise similarity signals whose ranking consistency degrades from ρs=0.88 to 0.27 in deep layers. Pairwise rankings are inherently unstable (O(Np2) joint perturbations) while unary signals enjoy greater stability (O(Np) perturbations, CLT). From three design principles derived from this diagnosis, we construct CATIS as a constructive validation: unary signals raise the trigger threshold, triage suppresses the gain. On ViT-Large at 63% FLOPs reduction, CATIS retains 96.9% of vanilla accuracy (81.0%) on ImageNet-1K where all baselines collapse to 43--65%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 50 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

42 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Accelerating Transformers with Spectrum-Preserving Token Merging25 May
  2. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  3. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers20 Jul
  4. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  5. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers21 Apr
  6. 2022Token Merging: Your ViT But Faster17 Oct
  7. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  8. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
22 more
  1. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  5. 2023Vision Transformers Need Registerscited by 871
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  7. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  8. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  9. 2021Intriguing Properties of Vision Transformerscited by 446
  10. 2022DeiT III: Revenge of the ViTcited by 419
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  13. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
  14. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  15. 2021Token Pooling in Vision Transformerscited by 69
  16. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 68
  17. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  18. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  19. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  20. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  21. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  22. 2024SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformerscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
  2. 2025Training-free Token Reduction for Vision Mamba
  3. 2025Lossless Token Merging Even Without Fine-Tuning in Vision Transformers
  4. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
  5. 2025Frequency-Aware Token Reduction for Efficient Vision Transformer
  6. 2026Later Is Better: Token Reduction for ViTs Under Distribution Shift
  7. 2026Stateful Token Reduction for Long-Video Hybrid VLMs
  8. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preserved