mlsanityData to 7 Oct 2026

DiffRate : Differentiable Compression Rate for Efficient Vision Transformers

arXiv 2305.17997 ↗First posted 29 May 2023cs.CVcited by 78

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Token compression aims to speed up large-scale vision transformers (e.g. ViTs) by pruning (dropping) or merging tokens. It is an important but challenging task. Although recent advanced approaches achieved great success, they need to carefully handcraft a compression rate (i.e. number of tokens to remove), which is tedious and leads to sub-optimal performance. To tackle this problem, we propose Differentiable Compression Rate (DiffRate), a novel token compression method that has several appealing properties prior arts do not have. First, DiffRate enables propagating the loss function's gradient onto the compression ratio, which is considered as a non-differentiable hyperparameter in previous work. In this case, different layers can automatically learn different compression rates layer-wisely without extra overhead. Second, token pruning and merging can be naturally performed simultaneously in DiffRate, while they were isolated in previous works. Third, extensive experiments demonstrate that DiffRate achieves state-of-the-art performance. For example, by applying the learned layer-wise compression rates to an off-the-shelf ViT-H (MAE) model, we achieve a 40% FLOPs reduction and a 1.5x throughput improvement, with a minor accuracy drop of 0.16% on ImageNet without fine-tuning, even outperforming previous methods with fine-tuning. Codes and models are available at https://github.com/OpenGVLab/DiffRate.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 47 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

38 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning11 Oct
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  4. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 34

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  11. 2020A Survey on Visual Transformercited by 653
  12. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
22 more
  1. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  2. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  3. 2021Post-Training Quantization for Vision Transformercited by 274
  4. 2022Masked Autoencoders that Listencited by 271
  5. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  6. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  7. 2023Token Merging for Fast Stable Diffusioncited by 220
  8. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  9. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  10. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  11. 2022MetaFormer Baselines for Visioncited by 131
  12. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  13. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 99
  14. 2020Operation-Aware Soft Channel Pruning using Differentiable Maskscited by 74
  15. 2020DMCP: Differentiable Markov Channel Pruning for Neural Networkscited by 68
  16. 2021Token Pooling in Vision Transformerscited by 68
  17. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  18. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  19. 2019Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integrationcited by 59
  20. 2021ISTR: End-to-End Instance Segmentation with Transformerscited by 52
  21. 2023You Only Segment Once: Towards Real-Time Panoptic Segmentationcited by 40
  22. 2022Super Vision Transformercited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 25; 2025: 23; 2026: 25.

78 papers.

Sort:

By area:

  1. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  2. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  3. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39
  4. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  5. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  6. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  7. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  8. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  9. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  10. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformerscited by 21
  11. 2025LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Modelscited by 21
  12. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
38 more
  1. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  2. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  3. 2025ToMA: Token Merge with Attention for Diffusion Modelscited by 17
  4. 2024MLP Can Be A Good Transformer Learnercited by 16
  5. 2023Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detectioncited by 13
  6. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  7. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  8. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  9. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTscited by 9
  10. 2024PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformercited by 8
  11. 2025HTTM: Head-wise Temporal Token Merging for Faster VGGTcited by 8
  12. 2024Efficient Time Series Processing for Transformers and State-Space Models through Token Mergingcited by 7
  13. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  14. 2025Not Every Patch is Needed: Towards a More Efficient and Effective Backbone for Video-based Person Re-identificationcited by 7
  15. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  16. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizabilitycited by 5
  17. 2024Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuningcited by 5
  18. 2024Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-trainingcited by 5
  19. 2025Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentationcited by 5
  20. 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
  21. 2025H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformerscited by 5
  22. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  23. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQAcited by 4
  24. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  25. 2024Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformerscited by 3
  26. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AIcited by 3
  27. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformerscited by 2
  28. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  29. 2025STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inferencecited by 2
  30. 2025Frequency-Aware Token Reduction for Efficient Vision Transformercited by 2
  31. 2026Shiva-DiT: Residual-Based Differentiable Top-k Selection for Efficient Diffusion Transformerscited by 2
  32. 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Modelscited by 2
  33. 2026GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstructioncited by 2
  34. 2023Bootstrapping SparseFormers from Vision Foundation Modelscited by 1
  35. 2024Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Modelscited by 1
  36. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Modelscited by 1
  37. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understandingcited by 1
  38. 2025Training-free Token Reduction for Vision Mambacited by 1

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preserved
  2. 2025Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration
  3. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
  4. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  5. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer
  6. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
  7. 2024TPC-ViT: Token Propagation Controller for Efficient Vision Transformer
  8. 2024Motion Guided Token Compression for Efficient Masked Video Modeling