mlsanityData to 7 Oct 2026

Zero Time Waste: Recycling Predictions in Early Exit Neural Networks

arXiv 2106.05409 ↗First posted 9 Jun 2021cs.LGcited by 39

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

The problem of reducing processing time of large deep learning models is a fundamental challenge in many real-world applications. Early exit methods strive towards this goal by attaching additional Internal Classifiers (ICs) to intermediate layers of a neural network. ICs can quickly return predictions for easy examples and, as a result, reduce the average inference time of the whole model. However, if a particular IC does not decide to return an answer early, its predictions are discarded, with its computations effectively being wasted. To solve this issue, we introduce Zero Time Waste (ZTW), a novel approach in which each IC reuses predictions returned by its predecessors by (1) adding direct connections between ICs and (2) combining previous outputs in an ensemble-like manner. We conduct extensive experiments across various datasets and architectures to demonstrate that ZTW achieves a significantly better accuracy vs. inference time trade-off than other recently proposed early exit methods.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 49 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

29 of 49 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2016Wide Residual Networkscited by 3,057
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  11. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
17 more
  1. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  2. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  3. 2019Challenges of Real-World Reinforcement Learningcited by 282
  4. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  5. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  6. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  7. 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
  8. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 99
  9. 2019A Comparative Analysis of XGBoostcited by 80
  10. 2020Why should we add early exits to neural networks?cited by 70
  11. 2019Improved Techniques for Training Adaptive Deep Networkscited by 66
  12. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 47
  13. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  14. 2017Deciding How to Decide: Dynamic Routing in Artificial Neural Networkscited by 33
  15. 2020PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learningcited by 30
  16. 2021GST: Group-Sparse Training for Accelerating Deep Reinforcement Learningcited by 7
  17. 2019Approximate LSTMs for Time-Constrained Inference: Enabling Fast Reaction in Self-Driving Carscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 12; 2024: 6; 2025: 10; 2026: 4.

39 papers.

Sort:

By area:

  1. 2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
  2. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  3. 2022Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inferencecited by 24
  4. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
  5. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
  6. 2023Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNNcited by 15
  7. 2022OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypothesescited by 11
  8. 2022Resource-Constrained Edge AI with Early Exit Predictioncited by 10
  9. 2024Conditional computation in neural networks: principles and research trendscited by 10
  10. 2024Exploring Dynamic Transformer for Efficient Object Trackingcited by 7
  11. 2023Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exitingcited by 6
  12. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
27 more
  1. 2021PluGeN: Multi-Label Conditional Generation From Pre-Trained Modelscited by 5
  2. 2023Risk-Controlling Model Selection via Guided Bayesian Optimizationcited by 5
  3. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  4. 2022Understanding the Robustness of Multi-Exit Models under Common Corruptionscited by 4
  5. 2023ScrollNet: Dynamic Weight Importance for Continual Learningcited by 4
  6. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  7. 2023Anticipate, Ensemble and Prune: Improving Convolutional Neural Networks via Aggregated Early Exitscited by 3
  8. 2023FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exitscited by 3
  9. 2023Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNscited by 3
  10. 2024On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performancecited by 3
  11. 2025You Only Look Once at Anytime (AnytimeYOLO): Analysis and Optimization of Early-Exits for Object-Detectioncited by 3
  12. 2025Energy-Aware Deep Learning on Resource-Constrained Hardwarecited by 3
  13. 2025Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliabilitycited by 3
  14. 2026Investigating Single-Block Recurrence in Vision Transformers for Image Recognitioncited by 3
  15. 2023TORE: Token Recycling in Vision Transformers for Efficient Active Visual Explorationcited by 2
  16. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  17. 2024Goal-oriented Communications based on Recursive Early Exit Neural Networkscited by 1
  18. 2025Onboard Optimization and Learning: A Surveycited by 1
  19. 2025FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inferencecited by 1
  20. 2025AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokenscited by 1
  21. 2022Towards Disentangling Information Paths with Coded ResNeXtcited by 0
  22. 2023Early Exiting Predictive Coding Neural Networks for Edge AIcited by 0
  23. 2025Know What You Don't Know: Selective Prediction for Early Exit DNNscited by 0
  24. 2025Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directionscited by 0
  25. 2026CRoSS: A Continual Robotic Simulation Suite for Scalable Reinforcement Learning with High Task Diversity and Realistic Physics Simulationcited by 0
  26. 2026ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptationcited by 0
  27. 2026SecondOpinion: Anatomy-Aware Gated Reasoning for Efficient Medical Image Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Early-Exit with Class Exclusion for Efficient Inference of Neural Networks
  2. 2022Pruning Early Exit Networks
  3. 2022Learning to Weight Samples for Dynamic Early-exiting Networks
  4. 2025Rethinking Calibration for Early-Exit Neural Networks
  5. 2024Early-exit Convolutional Neural Networks
  6. 2023Adaptive Deep Neural Network Inference Optimization with EENet
  7. 2022Unsupervised Early Exit in DNNs with Multiple Exits
  8. 2022Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices