# Zero Time Waste: Recycling Predictions in Early Exit Neural Networks

arXiv: https://arxiv.org/abs/2106.05409 · HTML: https://mlsanity.com/paper/2106.05409/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Conditional Computation in Neural Networks for faster models](https://mlsanity.com/paper/1511.06297/) (2015)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification](https://mlsanity.com/paper/2010.05300/) (2020)
- [A Comparative Analysis of XGBoost](https://mlsanity.com/paper/1911.01914/) (2019)
- [Why should we add early exits to neural networks?](https://mlsanity.com/paper/2004.12814/) (2020)
- [Improved Techniques for Training Adaptive Deep Networks](https://mlsanity.com/paper/1908.06294/) (2019)
- [IDK Cascades: Fast Deep Learning by Learning not to Overthink](https://mlsanity.com/paper/1706.00885/) (2017)
- [Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks](https://mlsanity.com/paper/1312.4461/) (2013)
- [Deciding How to Decide: Dynamic Routing in Artificial Neural Networks](https://mlsanity.com/paper/1703.06217/) (2017)
- [PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning](https://mlsanity.com/paper/2001.05012/) (2020)
- [GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09650/) (2021)
- [Approximate LSTMs for Time-Constrained Inference: Enabling Fast Reaction in Self-Driving Cars](https://mlsanity.com/paper/1905.00689/) (2019)

## Built on by (most cited first, 30 of 39)

- [LGViT: Dynamic Early Exiting for Accelerating Vision Transformer](https://mlsanity.com/paper/2308.00255/) (2023)
- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inference](https://mlsanity.com/paper/2206.04685/) (2022)
- [Efficiently Controlling Multiple Risks with Pareto Testing](https://mlsanity.com/paper/2210.07913/) (2022)
- [Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity](https://mlsanity.com/paper/2306.02652/) (2023)
- [Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNN](https://mlsanity.com/paper/2310.09163/) (2023)
- [OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses](https://mlsanity.com/paper/2204.02426/) (2022)
- [Resource-Constrained Edge AI with Early Exit Prediction](https://mlsanity.com/paper/2206.07269/) (2022)
- [Conditional computation in neural networks: principles and research trends](https://mlsanity.com/paper/2403.07965/) (2024)
- [Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting](https://mlsanity.com/paper/2308.16678/) (2023)
- [Exploring Dynamic Transformer for Efficient Object Tracking](https://mlsanity.com/paper/2403.17651/) (2024)
- [Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference](https://mlsanity.com/paper/2312.10193/) (2023)
- [PluGeN: Multi-Label Conditional Generation From Pre-Trained Models](https://mlsanity.com/paper/2109.09011/) (2021)
- [Risk-Controlling Model Selection via Guided Bayesian Optimization](https://mlsanity.com/paper/2312.01692/) (2023)
- [A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion](https://mlsanity.com/paper/2501.07451/) (2025)
- [Understanding the Robustness of Multi-Exit Models under Common Corruptions](https://mlsanity.com/paper/2212.01562/) (2022)
- [ScrollNet: Dynamic Weight Importance for Continual Learning](https://mlsanity.com/paper/2308.16567/) (2023)
- [RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference](https://mlsanity.com/paper/2405.15198/) (2024)
- [Investigating Single-Block Recurrence in Vision Transformers for Image Recognition](https://mlsanity.com/paper/2605.10661/) (2026)
- [Anticipate, Ensemble and Prune: Improving Convolutional Neural Networks via Aggregated Early Exits](https://mlsanity.com/paper/2301.12168/) (2023)
- [FIANCEE: Faster Inference of Adversarial Networks via Conditional Early Exits](https://mlsanity.com/paper/2304.10306/) (2023)
- [Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs](https://mlsanity.com/paper/2305.19844/) (2023)
- [On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2403.17154/) (2024)
- [You Only Look Once at Anytime (AnytimeYOLO): Analysis and Optimization of Early-Exits for Object-Detection](https://mlsanity.com/paper/2503.17497/) (2025)
- [Energy-Aware Deep Learning on Resource-Constrained Hardware](https://mlsanity.com/paper/2505.12523/) (2025)
- [Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability](https://mlsanity.com/paper/2509.23666/) (2025)
- [TORE: Token Recycling in Vision Transformers for Efficient Active Visual Exploration](https://mlsanity.com/paper/2311.15335/) (2023)
- [TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination](https://mlsanity.com/paper/2510.22767/) (2025)
- [Goal-oriented Communications based on Recursive Early Exit Neural Networks](https://mlsanity.com/paper/2412.19587/) (2024)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)

## Abstract (arXiv)

The problem of reducing processing time of large deep learning models is a fundamental challenge in many real-world applications. Early exit methods strive towards this goal by attaching additional Internal Classifiers (ICs) to intermediate layers of a neural network. ICs can quickly return predictions for easy examples and, as a result, reduce the average inference time of the whole model. However, if a particular IC does not decide to return an answer early, its predictions are discarded, with its computations effectively being wasted. To solve this issue, we introduce Zero Time Waste (ZTW), a novel approach in which each IC reuses predictions returned by its predecessors by (1) adding direct connections between ICs and (2) combining previous outputs in an ensemble-like manner. We conduct extensive experiments across various datasets and architectures to demonstrate that ZTW achieves a significantly better accuracy vs. inference time trade-off than other recently proposed early exit methods.