# Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions

arXiv: https://arxiv.org/abs/2106.05022 · HTML: https://mlsanity.com/paper/2106.05022/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 59 of its 88 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications](https://mlsanity.com/paper/1804.03230/) (2018)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [Dynamic Channel Pruning: Feature Boosting and Suppression](https://mlsanity.com/paper/1810.05331/) (2018)

## Built on by (most cited first, 30 of 75)

- [EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism](https://mlsanity.com/paper/2312.04916/) (2023)
- [Path Independent Equilibrium Models Can Better Exploit Test-Time Computation](https://mlsanity.com/paper/2211.09961/) (2022)
- [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024)
- [HMoE: Heterogeneous Mixture of Experts for Language Modeling](https://mlsanity.com/paper/2408.10681/) (2024)
- [Smart at what cost? Characterising Mobile Deep Neural Networks in the wild](https://mlsanity.com/paper/2109.13963/) (2021)
- [Mobile Foundation Model as Firmware](https://mlsanity.com/paper/2308.14363/) (2023)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration](https://mlsanity.com/paper/2411.02829/) (2024)
- [Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inference](https://mlsanity.com/paper/2206.04685/) (2022)
- [SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting](https://mlsanity.com/paper/2504.08850/) (2025)
- [Fixing Overconfidence in Dynamic Neural Networks](https://mlsanity.com/paper/2302.06359/) (2023)
- [AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments](https://mlsanity.com/paper/2303.07129/) (2023)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNN](https://mlsanity.com/paper/2310.09163/) (2023)
- [EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models](https://mlsanity.com/paper/2402.00518/) (2024)
- [Benchmarking of DL Libraries and Models on Mobile Devices](https://mlsanity.com/paper/2202.06512/) (2022)
- [The Future of Consumer Edge-AI Computing](https://mlsanity.com/paper/2210.10514/) (2022)
- [Resource-Constrained Edge AI with Early Exit Prediction](https://mlsanity.com/paper/2206.07269/) (2022)
- [Adaptive Deep Neural Network Inference Optimization with EENet](https://mlsanity.com/paper/2301.07099/) (2023)
- [Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation](https://mlsanity.com/paper/2407.03086/) (2024)
- [Maestro: Uncovering Low-Rank Structures via Trainable Decomposition](https://mlsanity.com/paper/2308.14929/) (2023)
- [Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting](https://mlsanity.com/paper/2308.16678/) (2023)
- [Adaptivity and Modularity for Efficient Generalization Over Task Complexity](https://mlsanity.com/paper/2310.08866/) (2023)
- [Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Experts](https://mlsanity.com/paper/2504.12988/) (2025)
- [Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs](https://mlsanity.com/paper/2209.13443/) (2022)
- [Federated Learning for Inference at Anytime and Anywhere](https://mlsanity.com/paper/2212.04084/) (2022)
- [Early-Exit Neural Networks with Nested Prediction Sets](https://mlsanity.com/paper/2311.05931/) (2023)
- [Benchmarking Ultra-Low-Power μNPUs](https://mlsanity.com/paper/2503.22567/) (2025)
- [Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networks](https://mlsanity.com/paper/2110.02865/) (2021)
- [Towards Practical Few-shot Federated NLP](https://mlsanity.com/paper/2212.00192/) (2022)

## Abstract (arXiv)

DNNs are becoming less and less over-parametrised due to recent advances in efficient model design, through careful hand-crafted or NAS-based methods. Relying on the fact that not all inputs require the same amount of computation to yield a confident prediction, adaptive inference is gaining attention as a prominent approach for pushing the limits of efficient deployment. Particularly, early-exit networks comprise an emerging direction for tailoring the computation depth of each input sample at runtime, offering complementary performance gains to other efficiency optimisations. In this paper, we decompose the design methodology of early-exit networks to its key components and survey the recent advances in each one of them. We also position early-exiting against other efficient inference solutions and provide our insights on the current challenges and most promising future directions for research in the field.