mlsanityData to 7 Oct 2026

Confident Adaptive Language Modeling

arXiv 2207.07061 ↗First posted 14 Jul 2022cs.CL cs.LGcited by 297

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments on three diverse text generation tasks, we demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to × 3 -- while provably maintaining high performance.

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 70 of its 91 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

70 of 91 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation3 Mar

Other references · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2015Teaching Machines to Read and Comprehendcited by 1,039
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2020Efficient Transformers: A Surveycited by 784
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  11. 2019Green AIcited by 620
  12. 2018Universal Transformerscited by 613
  13. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  14. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  15. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  16. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  17. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  18. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  19. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  20. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  21. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  22. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  23. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  24. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  25. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  26. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  27. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  28. 2019Depth-Adaptive Transformercited by 227
  29. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  30. 2022Memorizing Transformerscited by 219
  31. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  32. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  33. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  34. 2022Adaptive Conformal Predictions for Time Seriescited by 155
  35. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  36. 2022Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imagingcited by 107
  37. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  38. 2021The Efficiency Misnomercited by 98

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 43; 2024: 73; 2025: 67; 2026: 105.

297 papers.

Sort:

By area:

  1. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  3. 2022Efficiently Scaling Transformer Inferencecited by 529
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  6. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  7. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  8. 2022Conformal Risk Controlcited by 290
  9. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  10. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  12. 2023Speculative Decoding with Big Little Decodercited by 163
38 more
  1. 2023Conformal Language Modelingcited by 163
  2. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  6. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  7. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  8. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  9. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  10. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  11. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  12. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  13. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  14. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  15. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  16. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  17. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  18. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  19. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  20. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  21. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  22. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  23. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  24. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  25. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  26. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  27. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  28. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  29. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  30. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  31. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  32. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  33. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  34. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  35. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  36. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  37. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  38. 2024A Survey on Transformer Compressioncited by 44

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  2. 2025Accelerating Large Language Model Inference via Early-Exiting Algorithms
  3. 2024Improving Open-Ended Text Generation via Adaptive Decoding
  4. 2025SimLens for Early Exit in Large Language Models: Eliciting Accurate Latent Predictions with One More Token
  5. 2025Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference
  6. 2025Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration
  7. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  8. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules