mlsanityData to 7 Oct 2026

Elbert: Fast Albert with Confidence-Window Based Early Exit

arXiv 2107.00175 ↗First posted 1 Jul 2021cs.CLcited by 14

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Despite the great success in Natural Language Processing (NLP) area, large pre-trained language models like BERT are not well-suited for resource-constrained or real-time applications owing to the large number of parameters and slow inference speed. Recently, compressing and accelerating BERT have become important topics. By incorporating a parameter-sharing strategy, ALBERT greatly reduces the number of parameters while achieving competitive performance. Nevertheless, ALBERT still suffers from a long inference time. In this work, we propose the ELBERT, which significantly improves the average inference speed compared to ALBERT due to the proposed confidence-window based early exit mechanism, without introducing additional parameters or extra training overhead. Experimental results show that ELBERT achieves an adaptive inference speedup varying from 2× to 10× with negligible accuracy degradation compared to ALBERT on various datasets. Besides, ELBERT achieves higher accuracy than existing early exit methods used for accelerating BERT under the same computation cost. Furthermore, to understand the principle of the early exit mechanism, we also visualize the decision-making process of it in ELBERT.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 29 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

26 of 29 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
14 more
  1. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  3. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  4. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  5. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  6. 2019Revealing the Dark Secrets of BERTcited by 300
  7. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  8. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  10. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  11. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  12. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  13. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  14. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 3; 2024: 1; 2025: 3; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024Efficient Tuning and Inference for Large Language Models on Textual Graphscited by 54
  2. 2022Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformerscited by 28
  3. 2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
  4. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
  5. 2022BEBERT: Efficient and Robust Binary Ensemble BERTcited by 13
  6. 2023Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inferencecited by 9
  7. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depthcited by 9
  8. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognitioncited by 5
  9. 2023BERT Lost Patience Won't Be Robust to Adversarial Slowdowncited by 4
  10. 2025FedEL: Federated Elastic Learning for Heterogeneous Devicescited by 3
  11. 2022Latency Adjustable Transformer Encoder for Language Understandingcited by 1
  12. 2022Fast and Accurate FSA System Using ELBERT: An Efficient and Lightweight BERTcited by 1
2 more
  1. 2025EPEE: Towards Efficient and Effective Foundation Models in Biomedicinecited by 1
  2. 2025Efficient Large Language Models with Zero-Shot Adjustable Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
  2. 2021RomeBERT: Robust Training of Multi-Exit BERT
  3. 2024CEEBERT: Cross-Domain Inference in Early Exit BERT
  4. 2025NeoBERT: A Next-Generation BERT
  5. 2024DE3-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks
  6. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baseline
  7. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
  8. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exit