# CEEBERT: Cross-Domain Inference in Early Exit BERT

arXiv: https://arxiv.org/abs/2405.15039 · HTML: https://mlsanity.com/paper/2405.15039/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 23 May 2024.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [PonderNet: Learning to Ponder](https://mlsanity.com/paper/2107.05407/) (2021)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [Towards Efficient NLP: A Standard Evaluation and A Strong Baseline](https://mlsanity.com/paper/2110.07038/) (2021)
- [You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model](https://mlsanity.com/paper/2211.11152/) (2022)
- [A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation](https://mlsanity.com/paper/2203.01670/) (2022)
- [Controlling Computation versus Quality for Neural Sequence Models](https://mlsanity.com/paper/2002.07106/) (2020)
- [Train Once, Test Anywhere: Zero-Shot Learning for Text Classification](https://mlsanity.com/paper/1712.05972/) (2017)
- [Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning](https://mlsanity.com/paper/2111.00230/) (2021)
- [SplitEE: Early Exit in Deep Neural Networks with Split Computing](https://mlsanity.com/paper/2309.09195/) (2023)
- [Unsupervised Early Exit in DNNs with Multiple Exits](https://mlsanity.com/paper/2209.09480/) (2022)

## Built on by (most cited first, 12 of 12)

- [BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts](https://mlsanity.com/paper/2502.00745/) (2025)
- [DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs](https://mlsanity.com/paper/2410.04424/) (2024)
- [RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference](https://mlsanity.com/paper/2405.15198/) (2024)
- [Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approach](https://mlsanity.com/paper/2410.05338/) (2024)
- [Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability](https://mlsanity.com/paper/2509.23666/) (2025)
- [FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference](https://mlsanity.com/paper/2510.22641/) (2025)
- [DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving](https://mlsanity.com/paper/2511.20720/) (2025)
- [Distributed Inference on Mobile Edge and Cloud: A Data-Cartography based Clustering Approach](https://mlsanity.com/paper/2412.16616/) (2024)
- [Know What You Don't Know: Selective Prediction for Early Exit DNNs](https://mlsanity.com/paper/2509.11520/) (2025)
- [A Comparative Analysis on the Performance of Upper Confidence Bound Algorithms in Adaptive Deep Neural Networks](https://mlsanity.com/paper/2604.24810/) (2026)
- [Inconsistency-aware Multimodal Schr"odinger Bridge for Deepfake Localization](https://mlsanity.com/paper/2605.23113/) (2026)
- [SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference](https://mlsanity.com/paper/2608.13076/) (2026)

## Abstract (arXiv)

Pre-trained Language Models (PLMs), like BERT, with self-supervision objectives exhibit remarkable performance and generalization across various tasks. However, they suffer in inference latency due to their large size. To address this issue, side branches are attached at intermediate layers, enabling early inference of samples without requiring them to pass through all layers. However, the challenge is to decide which layer to infer and exit each sample so that the accuracy and latency are balanced. Moreover, the distribution of the samples to be inferred may differ from that used for training necessitating cross-domain adaptation. We propose an online learning algorithm named Cross-Domain Inference in Early Exit BERT (CeeBERT) that dynamically determines early exits of samples based on the level of confidence at each exit point. CeeBERT learns optimal thresholds from domain-specific confidence observed at intermediate layers on the fly, eliminating the need for labeled data. Experimental results on five distinct datasets with BERT and ALBERT models demonstrate CeeBERT's ability to improve latency by reducing unnecessary computations with minimal drop in performance. By adapting to the threshold values, CeeBERT can speed up the BERT/ALBERT models by 2× - 3.5× with minimal drop in accuracy.