mlsanityData to 7 Oct 2026

Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead

arXiv 2105.09121 ↗First posted 19 May 2021cs.LG cs.CVcited by 26

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 66 references that are in the map. First posted on arXiv 19 May 2021.

Builds on

43 of 66 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
31 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  5. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  6. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  9. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  10. 2021Dynamic Neural Networks: A Surveycited by 405
  11. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  12. 2018CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenescited by 267
  13. 2019Depth-Adaptive Transformercited by 227
  14. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  15. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  16. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  17. 2020Why should we add early exits to neural networks?cited by 70
  18. 2018Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Datacited by 69
  19. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  20. 2020Rethinking CNN Models for Audio Classificationcited by 48
  21. 2020Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inferencecited by 45
  22. 2020CNN-based Density Estimation and Crowd Counting: A Surveycited by 43
  23. 2016Automatic tagging using deep convolutional neural networkscited by 39
  24. 2017A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimationcited by 29
  25. 2019Highlight Every Step: Knowledge Distillation via Collaborative Teachingcited by 20
  26. 2020Ambient Sound Helps: Audiovisual Crowd Counting in Extreme Conditionscited by 13
  27. 2017Raw Waveform-based Audio Classification Using Sample-level CNN Architecturescited by 12
  28. 2019Dense Scale Network for Crowd Countingcited by 12
  29. 2021Improving the Accuracy of Early Exits in Multi-Exit Architectures via Curriculum Learningcited by 9
  30. 2017Improving Efficiency in Convolutional Neural Network with Multilinear Filterscited by 8
  31. 2021Local Critic Training for Model-Parallel Learning of Deep Neural Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 6; 2024: 2; 2025: 7; 2026: 4.

26 papers.

Sort:

By area:

  1. 2021Multi-Exit Vision Transformer for Dynamic Inferencecited by 28
  2. 2023On Optimal Caching and Model Multiplexing for Large Model Inferencecited by 20
  3. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  4. 2022Generalised Image Outpainting with U-Transformercited by 11
  5. 2022Multi-Tailed Vision Transformer for Efficient Inferencecited by 10
  6. 2023GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformercited by 10
  7. 2022Efficient High-Resolution Deep Learning: A Surveycited by 9
  8. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  9. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  10. 2022Graph Reinforcement Learning-based CNN Inference Offloading in Dynamic Edge Computingcited by 4
  11. 2026ViTNT-FIQA: Training-Free Face Image Quality Assessment with Vision Transformerscited by 4
  12. 2023PromptMix: Text-to-image diffusion models enhance the performance of lightweight networkscited by 3
14 more
  1. 2025You Only Look Once at Anytime (AnytimeYOLO): Analysis and Optimization of Early-Exits for Object-Detectioncited by 3
  2. 2022Analysis of the Effect of Low-Overhead Lossy Image Compression on the Performance of Visual Crowd Counting for Smart City Applicationscited by 2
  3. 2024Continual Low-Rank Scaled Dot-product Attentioncited by 2
  4. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  5. 2025LOBERT: Generative AI Foundation Model for Limit Order Book Messagescited by 2
  6. 2023Life Regression based Patch Slimming for Vision Transformerscited by 1
  7. 2025QPART: Adaptive Model Quantization and Dynamic Workload Balancing for Accuracy-aware Edge Inferencecited by 1
  8. 2025ACME: Adaptive Customization of Large Models via Distributed Systemscited by 1
  9. 2026SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observationcited by 1
  10. 2023Graph Self-Supervised Learning for Endoscopic Image Matchingcited by 0
  11. 2023Efficient Online Processing with Deep Neural Networkscited by 0
  12. 2024Image-Specific Adaptation of Transformer Encoders for Compute-Efficient Segmentationcited by 0
  13. 2026EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessmentcited by 0
  14. 2026ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformer
  2. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model
  3. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models
  4. 2026Accelerating Vision Foundation Models with Drop-in Depthwise Convolution
  5. 2026Amortized-Precision Quantization for Early-Exit Vision Transformers
  6. 2025FREE: Fast and Robust Vision Language Models with Early Exits
  7. 2023Dynamic Perceiver for Efficient Visual Recognition
  8. 2025Janus: Collaborative Vision Transformer Under Dynamic Network Environment