# Unsupervised Early Exit in DNNs with Multiple Exits

arXiv: https://arxiv.org/abs/2209.09480 · HTML: https://mlsanity.com/paper/2209.09480/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Why should we add early exits to neural networks?](https://mlsanity.com/paper/2004.12814/) (2020)
- [Towards Efficient NLP: A Standard Evaluation and A Strong Baseline](https://mlsanity.com/paper/2110.07038/) (2021)
- [Controlling Computation versus Quality for Neural Sequence Models](https://mlsanity.com/paper/2002.07106/) (2020)
- [Calibration-Aided Edge Inference Offloading via Adaptive Model Partitioning of Deep Neural Networks](https://mlsanity.com/paper/2010.16335/) (2020)
- [Early-exit deep neural networks for distorted images: providing an efficient edge offloading](https://mlsanity.com/paper/2108.09343/) (2021)
- [Online Algorithm for Unsupervised Sensor Selection](https://mlsanity.com/paper/1901.04676/) (2019)
- [Balancing Cost and Benefit with Tied-Multi Transformers](https://mlsanity.com/paper/2002.08614/) (2020)

## Built on by (most cited first, 10 of 10)

- [SplitEE: Early Exit in Deep Neural Networks with Split Computing](https://mlsanity.com/paper/2309.09195/) (2023)
- [CEEBERT: Cross-Domain Inference in Early Exit BERT](https://mlsanity.com/paper/2405.15039/) (2024)
- [I-SplitEE: Image classification in Split Computing DNNs with Early Exits](https://mlsanity.com/paper/2401.10541/) (2024)
- [CAPEEN: Image Captioning with Early Exits and Knowledge Distillation](https://mlsanity.com/paper/2410.04433/) (2024)
- [Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach](https://mlsanity.com/paper/2410.05338/) (2024)
- [Efficiency Robustness of Dynamic Deep Learning Systems](https://mlsanity.com/paper/2506.10831/) (2025)
- [SINF: Semantic Neural Network Inference with Semantic Subgraphs](https://mlsanity.com/paper/2310.01259/) (2023)
- [Distributed Inference on Mobile Edge and Cloud: A Data-Cartography based Clustering Approach](https://mlsanity.com/paper/2412.16616/) (2024)
- [A Comparative Analysis on the Performance of Upper Confidence Bound Algorithms in Adaptive Deep Neural Networks](https://mlsanity.com/paper/2604.24810/) (2026)
- [Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exit](https://mlsanity.com/paper/2609.12337/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) are generally designed as sequentially cascaded differentiable blocks/layers with a prediction module connected only to its last layer. DNNs can be attached with prediction modules at multiple points along the backbone where inference can stop at an intermediary stage without passing through all the modules. The last exit point may offer a better prediction error but also involves more computational resources and latency. An exit point that is `optimal' in terms of both prediction error and cost is desirable. The optimal exit point may depend on the latent distribution of the tasks and may change from one task type to another. During neural inference, the ground truth of instances may not be available and error rates at each exit point cannot be estimated. Hence one is faced with the problem of selecting the optimal exit in an unsupervised setting. Prior works tackled this problem in an offline supervised setting assuming that enough labeled data is available to estimate the error rate at each exit point and tune the parameters for better accuracy. However, pre-trained DNNs are often deployed in new domains for which a large amount of ground truth may not be available. We model the problem of exit selection as an unsupervised online learning problem and use bandit theory to identify the optimal exit point. Specifically, we focus on Elastic BERT, a pre-trained multi-exit DNN to demonstrate that it `nearly' satisfies the Strong Dominance (SD) property making it possible to learn the optimal exit in an online setup without knowing the ground truth labels. We develop upper confidence bound (UCB) based algorithm named UEE-UCB that provably achieves sub-linear regret under the SD property. Thus our method provides a means to adaptively learn domain-specific optimal exit points in multi-exit DNNs. We empirically validate our algorithm on IMDb and Yelp datasets.