# Multi-Exit Vision Transformer for Dynamic Inference

arXiv: https://arxiv.org/abs/2106.15183 · HTML: https://mlsanity.com/paper/2106.15183/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Why should we add early exits to neural networks?](https://mlsanity.com/paper/2004.12814/) (2020)
- [TransCrowd: weakly-supervised crowd counting with transformers](https://mlsanity.com/paper/2104.09116/) (2021)
- [Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference](https://mlsanity.com/paper/2002.10025/) (2020)
- [CNN-based Density Estimation and Crowd Counting: A Survey](https://mlsanity.com/paper/2003.12783/) (2020)
- [Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead](https://mlsanity.com/paper/2105.09121/) (2021)
- [Ambient Sound Helps: Audiovisual Crowd Counting in Extreme Conditions](https://mlsanity.com/paper/2005.07097/) (2020)
- [Dense Scale Network for Crowd Counting](https://mlsanity.com/paper/1906.09707/) (2019)
- [Improving the Accuracy of Early Exits in Multi-Exit Architectures via Curriculum Learning](https://mlsanity.com/paper/2104.10461/) (2021)
- [Can Attention Enable MLPs To Catch Up With CNNs?](https://mlsanity.com/paper/2105.15078/) (2021)
- [Temporal Early Exits for Efficient Video Object Detection](https://mlsanity.com/paper/2106.11208/) (2021)

## Built on by (most cited first, 28 of 28)

- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [LGViT: Dynamic Early Exiting for Accelerating Vision Transformer](https://mlsanity.com/paper/2308.00255/) (2023)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking](https://mlsanity.com/paper/2503.06625/) (2025)
- [Dynamic Split Computing for Efficient Deep Edge Intelligence](https://mlsanity.com/paper/2205.11269/) (2022)
- [Continual Transformers: Redundancy-Free Attention for Online Inference](https://mlsanity.com/paper/2201.06268/) (2022)
- [Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importance](https://mlsanity.com/paper/2504.01690/) (2025)
- [Multi-Tailed Vision Transformer for Efficient Inference](https://mlsanity.com/paper/2203.01587/) (2022)
- [Efficient High-Resolution Deep Learning: A Survey](https://mlsanity.com/paper/2207.13050/) (2022)
- [HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.10724/) (2025)
- [ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference](https://mlsanity.com/paper/2507.10800/) (2025)
- [Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models](https://mlsanity.com/paper/2411.04291/) (2024)
- [ViTNT-FIQA: Training-Free Face Image Quality Assessment with Vision Transformers](https://mlsanity.com/paper/2601.05741/) (2026)
- [You Only Look Once at Anytime (AnytimeYOLO): Analysis and Optimization of Early-Exits for Object-Detection](https://mlsanity.com/paper/2503.17497/) (2025)
- [EERO: Early Exit with Reject Option for Efficient Classification with limited budget](https://mlsanity.com/paper/2402.03779/) (2024)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Learning Motion Blur Robust Vision Transformers for Real-Time UAV Tracking](https://mlsanity.com/paper/2407.05383/) (2024)
- [Efficiency Robustness of Dynamic Deep Learning Systems](https://mlsanity.com/paper/2506.10831/) (2025)
- [ACME: Adaptive Customization of Large Models via Distributed Systems](https://mlsanity.com/paper/2507.14802/) (2025)
- [Layer-Guided UAV Tracking: Enhancing Efficiency and Occlusion Robustness](https://mlsanity.com/paper/2602.13636/) (2026)
- [Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing](https://mlsanity.com/paper/2211.13778/) (2022)
- [Image-Specific Adaptation of Transformer Encoders for Compute-Efficient Segmentation](https://mlsanity.com/paper/2404.15244/) (2024)
- [A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits](https://mlsanity.com/paper/2604.14789/) (2026)
- [EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment](https://mlsanity.com/paper/2604.22842/) (2026)
- [Progressive Approximation in Deep Residual Networks: Theory and Validation](https://mlsanity.com/paper/2604.24154/) (2026)
- [Amortized-Precision Quantization for Early-Exit Vision Transformers](https://mlsanity.com/paper/2605.07317/) (2026)
- [The Rescue Effect: Spatio-Semantic Early Exit Bypasses Quantization Collapse in CLIP](https://mlsanity.com/paper/2605.26415/) (2026)
- [ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation](https://mlsanity.com/paper/2606.12023/) (2026)

## Abstract (arXiv)

Deep neural networks can be converted to multi-exit architectures by inserting early exit branches after some of their intermediate layers. This allows their inference process to become dynamic, which is useful for time critical IoT applications with stringent latency requirements, but with time-variant communication and computation resources. In particular, in edge computing systems and IoT networks where the exact computation time budget is variable and not known beforehand. Vision Transformer is a recently proposed architecture which has since found many applications across various domains of computer vision. In this work, we propose seven different architectures for early exit branches that can be used for dynamic inference in Vision Transformer backbones. Through extensive experiments involving both classification and regression problems, we show that each one of our proposed architectures could prove useful in the trade-off between accuracy and speed.