# Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity

arXiv: https://arxiv.org/abs/2306.02652 · HTML: https://mlsanity.com/paper/2306.02652/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2105.15075/) (2021)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges](https://mlsanity.com/paper/2103.04505/) (2021)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [Multimodal Conditional Image Synthesis with Product-of-Experts GANs](https://mlsanity.com/paper/2112.05130/) (2021)
- [Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions](https://mlsanity.com/paper/1410.7827/) (2014)
- [Improved Techniques for Training Adaptive Deep Networks](https://mlsanity.com/paper/1908.06294/) (2019)
- [Depth Uncertainty in Neural Networks](https://mlsanity.com/paper/2006.08437/) (2020)
- [Learning to Weight Samples for Dynamic Early-exiting Networks](https://mlsanity.com/paper/2209.08310/) (2022)
- [Zero Time Waste: Recycling Predictions in Early Exit Neural Networks](https://mlsanity.com/paper/2106.05409/) (2021)
- [A Survey on Dynamic Neural Networks for Natural Language Processing](https://mlsanity.com/paper/2202.07101/) (2022)
- [Fixing Overconfidence in Dynamic Neural Networks](https://mlsanity.com/paper/2302.06359/) (2023)
- [Anytime Dense Prediction with Confidence Adaptivity](https://mlsanity.com/paper/2104.00749/) (2021)
- [Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation](https://mlsanity.com/paper/2107.04212/) (2021)
- [Learning from demonstration using products of experts: applications to manipulation and task prioritization](https://mlsanity.com/paper/2010.03505/) (2020)
- [State-space Abstraction for Anytime Evaluation of Probabilistic Networks](https://mlsanity.com/paper/1302.6850/) (2013)

## Built on by (most cited first, 16 of 16)

- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [Cascade-Aware Training of Language Models](https://mlsanity.com/paper/2406.00060/) (2024)
- [On the Role of Depth and Looping for In-Context Learning with Task Diversity](https://mlsanity.com/paper/2410.21698/) (2024)
- [Language Models Can Predict Their Own Behavior](https://mlsanity.com/paper/2502.13329/) (2025)
- [Early-Exit Neural Networks with Nested Prediction Sets](https://mlsanity.com/paper/2311.05931/) (2023)
- [Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones](https://mlsanity.com/paper/2412.19325/) (2024)
- [QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring](https://mlsanity.com/paper/2404.12599/) (2024)
- [RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference](https://mlsanity.com/paper/2405.15198/) (2024)
- [Dynamic Vocabulary Pruning in Early-Exit LLMs](https://mlsanity.com/paper/2410.18952/) (2024)
- [Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting](https://mlsanity.com/paper/2510.02480/) (2025)
- [DuoDiff: Accelerating Diffusion Models with a Dual-Backbone Approach](https://mlsanity.com/paper/2410.09633/) (2024)
- [Rethinking Calibration for Early-Exit Neural Networks](https://mlsanity.com/paper/2508.21495/) (2025)
- [TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML](https://mlsanity.com/paper/2508.12905/) (2025)
- [RRFC: Recursive Refinement via Feedback Conditioning for Iterative Image-to-Image Generation](https://mlsanity.com/paper/2608.15694/) (2026)
- [Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement](https://mlsanity.com/paper/2609.29867/) (2026)

## Abstract (arXiv)

Modern predictive models are often deployed to environments in which computational budgets are dynamic. Anytime algorithms are well-suited to such environments as, at any point during computation, they can output a prediction whose quality is a function of computation time. Early-exit neural networks have garnered attention in the context of anytime computation due to their capability to provide intermediate predictions at various stages throughout the network. However, we demonstrate that current early-exit networks are not directly applicable to anytime settings, as the quality of predictions for individual data points is not guaranteed to improve with longer computation. To address this shortcoming, we propose an elegant post-hoc modification, based on the Product-of-Experts, that encourages an early-exit network to become gradually confident. This gives our deep models the property of conditional monotonicity in the prediction quality -- an essential stepping stone towards truly anytime predictive modeling using early-exit architectures. Our empirical results on standard image-classification tasks demonstrate that such behaviors can be achieved while preserving competitive accuracy on average.