mlsanityData to 7 Oct 2026

ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference

arXiv 2312.11882 ↗First posted 19 Dec 2023cs.CL cs.AI cs.LGcited by 13

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Early Exiting is one of the most popular methods to achieve efficient inference. Current early exiting methods adopt the (weighted) sum of the cross entropy loss of all internal classifiers during training, imposing all these classifiers to predict all instances correctly. However, during inference, as long as one internal classifier predicts an instance correctly, it can accelerate without losing accuracy. Thus, there is a notable gap between training and inference. We propose ConsistentEE, an early exiting method that is consistent in training and inference. ConsistentEE formulates the early exiting process as a reinforcement learning problem. A policy network is added to decide whether an instance should exit or continue. The training objective of ConsistentEE only require each instance to be predicted correctly by one internal classifier. Additionally, we introduce the concept Memorize Layer to measure the hardness of an instance. We incorporate memorized layer into reward function design, which allows "easy" instances to focus more on acceleration while "hard" instances to focus more on accuracy. Experimental results show that our method outperforms other baselines on various natural language understanding and generation tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 50 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

31 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation3 Mar
  2. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference25 May

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2017A Closer Look at Memorization in Deep Networkscited by 936
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
17 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  7. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  8. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  10. 2019Depth-Adaptive Transformercited by 227
  11. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  12. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  13. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  14. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  15. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  16. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  17. 2021RomeBERT: Robust Training of Multi-Exit BERTcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  3. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
  4. 2024DE3-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networkscited by 3
  5. 2024Dynamic layer selection in decoder-only transformerscited by 3
  6. 2025GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generationcited by 3
  7. 2025Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projectioncited by 2
  8. 2024COSEE: Consistency-Oriented Signal-Based Early Exiting via Calibrated Sample Weighting Mechanismcited by 1
  9. 2025Hetero-SplitEE: Split Learning of Neural Networks with Early Exits for Heterogeneous IoT Devicescited by 0
  10. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  11. 2026Two-dimensional early exit optimisation of LLM inferencecited by 0
  12. 2026N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generationcited by 0
1 more
  1. 2026FrankenReport: Early Exiting in Long-Form Generation Using Expected Value of Computationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inference
  2. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learning
  3. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  4. 2024An Efficient Inference Framework for Early-exit Large Language Models
  5. 2026Probe the Harness: Setup Checks for Stale-Data RL Comparisons in Language Models
  6. 2026ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment
  7. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
  8. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight