mlsanityData to 14 Oct 2026

SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer

arXiv 2303.17605 ↗First posted 30 Mar 2023cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

High-resolution images enable neural networks to learn richer visual representations. However, this improved performance comes at the cost of growing computational complexity, hindering their usage in latency-sensitive applications. As not all pixels are equal, skipping computations for less-important regions offers a simple and effective measure to reduce the computation. This, however, is hard to be translated into actual speedup for CNNs since it breaks the regularity of the dense convolution workload. In this paper, we introduce SparseViT that revisits activation sparsity for recent window-based vision transformers (ViTs). As window attentions are naturally batched over blocks, actual speedup with window activation pruning becomes possible: i.e., ~50% latency reduction with 60% sparsity. Different layers should be assigned with different pruning ratios due to their diverse sensitivities and computational costs. We introduce sparsity-aware adaptation and apply the evolutionary search to efficiently find the optimal layerwise sparsity configuration within the vast search space. SparseViT achieves speedups of 1.5x, 1.4x, and 1.3x compared to its dense counterpart in monocular 3D object detection, 2D instance segmentation, and 2D semantic segmentation, respectively, with negligible to no loss of accuracy.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2017Mask R-CNNcited by 7,277
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
38 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  7. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  9. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  13. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  14. 2021Transformers in Vision: A Surveycited by 1,013
  15. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  16. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  17. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  18. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  19. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  20. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  21. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  22. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  23. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  24. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  25. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  26. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  27. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  28. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  29. 2021Mobile-Former: Bridging MobileNet and Transformercited by 235
  30. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  31. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  32. 2021Learned Token Pruning for Transformerscited by 139
  33. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  34. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  35. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  36. 2022TorchSparse: Efficient Point Cloud Inference Enginecited by 73
  37. 2018SBNet: Sparse Blocks Network for Fast Inferencecited by 54
  38. 2020Sparse Weight Activation Trainingcited by 45

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 21; 2025: 20; 2026: 12.

60 papers.

Sort:

By area:

  1. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  2. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  3. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 62
  4. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  5. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  6. 2024Scene Adaptive Sparse Transformer for Event-based Object Detectioncited by 39
  7. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  8. 2024Restore-RWKV: Efficient and Effective Medical Image Restoration with RWKVcited by 36
  9. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 28
  10. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 16
  11. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  12. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
38 more
  1. 2024Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategycited by 14
  2. 2024Agglomerative Token Clusteringcited by 13
  3. 2025Accelerating Vision Transformers with Adaptive Patch Sizescited by 13
  4. 2023Win-Win: Training High-Resolution Vision Transformers from Two Windowscited by 12
  5. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  6. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Modelscited by 11
  7. 2024Conditional computation in neural networks: principles and research trendscited by 10
  8. 2024Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptationcited by 10
  9. 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 10
  10. 2025HTTM: Head-wise Temporal Token Merging for Faster VGGTcited by 9
  11. 2025RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancycited by 7
  12. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compressioncited by 7
  13. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 6
  14. 2024MaskVD: Region Masking for Efficient Video Object Detectioncited by 5
  15. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Modelscited by 5
  16. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  17. 2025IPCV: Information-Preserving Compression for MLLM Visual Encoderscited by 5
  18. 2023SparseSwin: Swin Transformer with Sparse Transformer Blockcited by 4
  19. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  20. 2024Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Headscited by 3
  21. 2024Sparse Refinement for Efficient High-Resolution Semantic Segmentationcited by 3
  22. 2025MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspectivecited by 3
  23. 2024DM3D: Distortion-Minimized Weight Pruning for Lossless 3D Object Detectioncited by 2
  24. 2024HCPM: Hierarchical Candidates Pruning for Efficient Detector-Free Matchingcited by 1
  25. 2024Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learningcited by 1
  26. 2024Context-Aware Token Selection and Packing for Enhanced Vision Transformercited by 1
  27. 2025Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphscited by 1
  28. 2025AMMSM: Adaptive Motion Magnification and Sparse Mamba for Micro-Expression Recognitioncited by 1
  29. 2025Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detectioncited by 1
  30. 2026Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspectivecited by 1
  31. 2023Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputscited by 0
  32. 2025vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognitioncited by 0
  33. 2025TMT: Cross-domain Semantic Segmentation with Region-adaptive Transferability Estimationcited by 0
  34. 2025SPoT: Subpixel Placement of Tokens in Vision Transformerscited by 0
  35. 2025PSTTS: A Plug-and-Play Token Selector for Efficient Event-based Spatio-temporal Representation Learningcited by 0
  36. 2025DinoLizer: Separating VAE and Diffusion Artifacts in Generative Inpainting Localizationcited by 0
  37. 2025Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformerscited by 0
  38. 2026ViT Registers and Fractal ViTcited by 0

50 most cited of 60 All 60 (JSON)