mlsanityData to 7 Oct 2026

Dense Vision Transformer Compression with Few Samples

arXiv 2403.18708 ↗First posted 27 Mar 2024cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Few-shot model compression aims to compress a large model into a more compact one with only a tiny training set (even without labels). Block-level pruning has recently emerged as a leading technique in achieving high accuracy and low latency in few-shot CNN compression. But, few-shot compression for Vision Transformers (ViT) remains largely unexplored, which presents a new challenge. In particular, the issue of sparse compression exists in traditional CNN few-shot methods, which can only produce very few compressed models of different model sizes. This paper proposes a novel framework for few-shot ViT compression named DC-ViT. Instead of dropping the entire block, DC-ViT selectively eliminates the attention module while retaining and reusing portions of the MLP module. DC-ViT enables dense compression, which outputs numerous compressed models that densely populate the range of model complexity. DC-ViT outperforms state-of-the-art few-shot compression methods by a significant margin of 10 percentage points, along with lower latency in the compression of ViT and its variants.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

43 of 59 references.

References · 43

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  12. 2016Pruning Filters for Efficient ConvNetscited by 1,129
31 more
  1. 2020Designing Network Design Spacescited by 954
  2. 2020Pre-Trained Image Processing Transformercited by 891
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  4. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  5. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  6. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  7. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  8. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  9. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  10. 2018Video Action Transformer Networkcited by 316
  11. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  12. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  13. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  14. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  15. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  16. 2018Delta-encoder: an effective sample synthesis method for few-shot object recognitioncited by 89
  17. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  18. 2022Unified Visual Transformer Compressioncited by 79
  19. 2021Vision Transformer Pruningcited by 77
  20. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  21. 2018Multi-level Semantic Feature Augmentation for One-shot Learningcited by 62
  22. 2020DADA: Differentiable Automatic Data Augmentationcited by 57
  23. 2019Image Deformation Meta-Networks for One-Shot Learningcited by 56
  24. 2019Few-Shot Learning via Saliency-guided Hallucination of Samplescited by 49
  25. 2019LaSO: Label-Set Operations networks for multi-label few-shot learningcited by 38
  26. 2018Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networkscited by 35
  27. 2019Few Shot Network Compression via Cross Distillationcited by 26
  28. 2019DBP: Discrimination Based Block-Level Pruning for Deep Model Accelerationcited by 24
  29. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  30. 2022Practical Network Acceleration with Tiny Setscited by 9
  31. 2022Compressing Models with Few Samples: Mimicking then Replacingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 3.

8 papers.

  1. 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approachcited by 8
  2. 2025RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layerscited by 5
  3. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  4. 2024TOAST: Transformer Optimization using Adaptive and Simple Transformationscited by 1
  5. 2026TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformerscited by 1
  6. 2025NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devicescited by 0
  7. 2026CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformerscited by 0
  8. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0