Data to 7 Oct 2026

BERT Loses Patience: Fast and Robust Inference with Early Exit

arXiv 2006.04152 ↗First posted 7 Jun 2020cs.CL cs.LGcited by 267

Not on the map. Why?

Cited by 85 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM). To achieve this, our approach couples an internal-classifier with each layer of a PLM and dynamically stops inference when the intermediate predictions of the internal classifiers remain unchanged for a pre-defined number of steps. Our approach improves inference efficiency as it allows the model to make a prediction with fewer layers. Meanwhile, experimental results with an ALBERT model show that our method can improve the accuracy and robustness of the model by preventing it from overthinking and exploiting multiple classifiers for prediction, yielding a better accuracy-speed trade-off compared to existing early exit methods.

Key facts. Cited by 267 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 39; 2023: 57; 2024: 52; 2025: 46; 2026: 42.

267 papers.

Sort:

By area:

2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
2021Dynamic Neural Networks: A Surveycited by 405
2021A Survey of Transformerscited by 399
2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
2022Confident Adaptive Language Modelingcited by 297
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
2023AutoMix: Automatically Mixing Language Modelscited by 116
2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111

38 more

2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
2025Universal Model Routing for Efficient LLM Inferencecited by 80
2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
2024Efficient Tuning and Inference for Large Language Models on Textual Graphscited by 54
2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
2021A Survey on Green Deep Learningcited by 52
2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 29

50 most cited of 267 All 267 (JSON)