mlsanityData to 7 Oct 2026

Accelerating BERT Inference for Sequence Labeling via Early-Exit

arXiv 2105.13878 ↗First posted 28 May 2021cs.CL cs.AIcited by 29

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Both performance and efficiency are crucial factors for sequence labeling tasks in many real-world scenarios. Although the pre-trained models (PTMs) have significantly improved the performance of various sequence labeling tasks, their computational cost is expensive. To alleviate this problem, we extend the recent successful early-exit mechanism to accelerate the inference of PTMs for sequence labeling tasks. However, existing early-exit mechanisms are specifically designed for sequence-level tasks, rather than sequence labeling. In this paper, we first propose a simple extension of sentence-level early-exit for sequence labeling tasks. To further reduce the computational cost, we also propose a token-level early-exit mechanism that allows partial tokens to exit early at different layers. Considering the local dependency inherent in sequence labeling, we employed a window-based criterion to decide for a token whether or not to exit. The token-level early-exit brings the gap between training and inference, so we introduce an extra self-sampling fine-tuning stage to alleviate it. The extensive experiments on three popular sequence labeling tasks show that our approach can save up to 66%-75% inference cost with minimal performance degradation. Compared with competitive compressed models such as DistilBERT, our approach can achieve better performance under the same speed-up ratios of 2X, 3X, and 4X.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 44 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

25 of 44 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
  9. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  11. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  12. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
13 more
  1. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  2. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  3. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  4. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  5. 2019Depth-Adaptive Transformercited by 227
  6. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  7. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  8. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  9. 2019TENER: Adapting Transformer Encoder for Named Entity Recognitioncited by 69
  10. 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
  11. 2020Coarse-to-Fine Pre-training for Named Entity Recognitioncited by 28
  12. 2016F-Score Driven Max Margin Neural Network for Named Entity Recognition in Chinese Social Mediacited by 11
  13. 2020Unified Multi-Criteria Chinese Word Segmentation with BERTcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 5; 2024: 10; 2025: 3; 2026: 3.

29 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 399
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  4. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  5. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  6. 2021A Survey on Green Deep Learningcited by 52
  7. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  8. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  9. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  10. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  11. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  12. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
17 more
  1. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  2. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  3. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  4. 2023Dynamic Transformers Provide a False Sense of Efficiencycited by 11
  5. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10
  6. 2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
  7. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Modelscited by 7
  8. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  9. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  10. 2024DE3-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networkscited by 3
  11. 2024CAPEEN: Image Captioning with Early Exits and Knowledge Distillationcited by 3
  12. 2025Position-Aware Depth Decay Decoding (D3): Boosting Large Language Model Inference Efficiencycited by 2
  13. 2026SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibrationcited by 1
  14. 2024C3: Confidence Calibration Model Cascade for Inference-Efficient Cross-Lingual Natural Language Understandingcited by 0
  15. 2025Nav-EE: Navigation-Guided Early Exiting for Efficient Vision-Language Models in Autonomous Drivingcited by 0
  16. 2026FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessmentcited by 0
  17. 2026Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
  2. 2022Token Dropping for Efficient BERT Pretraining
  3. 2024CEEBERT: Cross-Domain Inference in Early Exit BERT
  4. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
  5. 2024An Efficient Inference Framework for Early-exit Large Language Models
  6. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretraining
  7. 2021Elbert: Fast Albert with Confidence-Window Based Early Exit
  8. 2021Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning