mlsanityData to 7 Oct 2026

Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions

arXiv 2106.05022 ↗First posted 9 Jun 2021cs.LGcited by 74

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

DNNs are becoming less and less over-parametrised due to recent advances in efficient model design, through careful hand-crafted or NAS-based methods. Relying on the fact that not all inputs require the same amount of computation to yield a confident prediction, adaptive inference is gaining attention as a prominent approach for pushing the limits of efficient deployment. Particularly, early-exit networks comprise an emerging direction for tailoring the computation depth of each input sample at runtime, offering complementary performance gains to other efficiency optimisations. In this paper, we decompose the design methodology of early-exit networks to its key components and survey the recent advances in each one of them. We also position early-exiting against other efficient inference solutions and provide our insights on the current challenges and most promising future directions for research in the field.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 59 of its 88 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

59 of 88 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges8 Mar

Other references · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
38 more
  1. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  2. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  3. 2021Dynamic Neural Networks: A Surveycited by 405
  4. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  5. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  6. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  7. 2019Depth-Adaptive Transformercited by 227
  8. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  9. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  10. 2018To Trust Or Not To Trust A Classifiercited by 205
  11. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  12. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  13. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 150
  14. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  15. 2018NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applicationscited by 146
  16. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  17. 2017Deep Learning for Sensor-based Activity Recognition: A Surveycited by 133
  18. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  19. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  20. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 109
  21. 2020Temporally Distributed Networks for Fast Video Semantic Segmentationcited by 77
  22. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  23. 2018NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Visioncited by 68
  24. 2015Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognitioncited by 67
  25. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  26. 2017Do Convolutional Neural Networks Learn Class Hierarchy?cited by 61
  27. 2020A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inferencecited by 61
  28. 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
  29. 2017Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascadecited by 53
  30. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 47
  31. 2018You Look Twice: GaterNet for Dynamic Filter Selection in CNNscited by 46
  32. 2020Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inferencecited by 45
  33. 2019Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inferencecited by 31
  34. 2020Learning to Stop While Learning to Predictcited by 30
  35. 2020HAPI: Hardware-Aware Progressive Inferencecited by 30
  36. 2016Feedback Networkscited by 29
  37. 2020Early Exit or Not: Resource-Efficient Blind Quality Enhancement for Compressed Imagescited by 28
  38. 2019MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processorscited by 27

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 12; 2023: 20; 2024: 14; 2025: 16; 2026: 8.

74 papers.

Sort:

By area:

  1. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  2. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  3. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
  4. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  5. 2021Smart at what cost? Characterising Mobile Deep Neural Networks in the wildcited by 25
  6. 2023Mobile Foundation Model as Firmwarecited by 25
  7. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  8. 2022Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inferencecited by 24
  9. 2023Green Edge AI: A Contemporary Surveycited by 24
  10. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  11. 2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
  12. 2023AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environmentscited by 19
38 more
  1. 2023Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Modelscited by 15
  2. 2023Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNNcited by 15
  3. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  4. 2022Benchmarking of DL Libraries and Models on Mobile Devicescited by 12
  5. 2022The Future of Consumer Edge-AI Computingcited by 12
  6. 2022Resource-Constrained Edge AI with Early Exit Predictioncited by 10
  7. 2023Adaptive Deep Neural Network Inference Optimization with EENetcited by 9
  8. 2024Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generationcited by 9
  9. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  10. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  11. 2022Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUscited by 6
  12. 2023Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exitingcited by 6
  13. 2023Early-Exit Neural Networks with Nested Prediction Setscited by 6
  14. 2025Benchmarking Ultra-Low-Power μNPUscited by 6
  15. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
  16. 2021Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networkscited by 5
  17. 2022Towards Practical Few-shot Federated NLPcited by 5
  18. 2022Federated Learning for Inference at Anytime and Anywherecited by 5
  19. 2023Risk-Controlling Model Selection via Guided Bayesian Optimizationcited by 5
  20. 2022Understanding the Robustness of Multi-Exit Models under Common Corruptionscited by 4
  21. 2024Tiny Models are the Computational Saver for Large Modelscited by 4
  22. 2024Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Onescited by 4
  23. 2025Multi-Exit Kolmogorov-Arnold Networks: enhancing accuracy and parsimonycited by 4
  24. 2022Multi-DNN Accelerators for Next-Generation AI Systemscited by 3
  25. 2023ATHEENA: A Toolflow for Hardware Early-Exit Network Automationcited by 3
  26. 2023FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exitscited by 3
  27. 2023Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolutioncited by 3
  28. 2023RAI4IoE: Responsible AI for Enabling the Internet of Energycited by 3
  29. 2024LightFF: Lightweight Inference for Forward-Forward Algorithmcited by 3
  30. 2025You Only Look Once at Anytime (AnytimeYOLO): Analysis and Optimization of Early-Exits for Object-Detectioncited by 3
  31. 2025Energy-Aware Deep Learning on Resource-Constrained Hardwarecited by 3
  32. 2025TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precisioncited by 3
  33. 2021How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architecturescited by 2
  34. 2024EERO: Early Exit with Reject Option for Efficient Classification with limited budgetcited by 2
  35. 2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
  36. 2022Binary Early-Exit Network for Adaptive Inference on Low-Resource Devicescited by 1
  37. 2023Sense, Predict, Adapt, Repeat: A Blueprint for Design of New Adaptive AI-Centric Sensing Systemscited by 1
  38. 2025Onboard Optimization and Learning: A Surveycited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Early Exit Deep Neural Networks in NLP
  2. 2026Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks
  3. 2022Learning to Weight Samples for Dynamic Early-exiting Networks
  4. 2024Early-Exit meets Model-Distributed Inference at Edge Networks
  5. 2023Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings
  6. 2023Adaptive Early Exiting for Collaborative Inference over Noisy Wireless Channels
  7. 2023Early-Exit with Class Exclusion for Efficient Inference of Neural Networks
  8. 2024Fast yet Safe: Early-Exiting with Risk Control