mlsanityData to 7 Oct 2026

Consistent Accelerated Inference via Confident Adaptive Transformers

arXiv 2104.08803 ↗First posted 18 Apr 2021cs.CL cs.AI cs.LGcited by 73

Not on the map. Why?

Cited by 22 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We develop a novel approach for confidently accelerating inference in the large and expensive multilayer Transformers that are now ubiquitous in natural language processing (NLP). Amortized or approximate computational methods increase efficiency, but can come with unpredictable performance costs. In this work, we present CATs -- Confident Adaptive Transformers -- in which we simultaneously increase computational efficiency, while guaranteeing a specifiable degree of consistency with the original model with high confidence. Our method trains additional prediction heads on top of intermediate layers, and dynamically decides when to stop allocating computational effort to each input using a meta consistency classifier. To calibrate our early prediction stopping rule, we formulate a unique extension of conformal prediction. We demonstrate the effectiveness of this approach on four classification and regression tasks.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

34 of 57 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
22 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2017Selective Classification for Deep Neural Networkscited by 799
  4. 2018Universal Transformerscited by 613
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  6. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  7. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  8. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  9. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  10. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  11. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  12. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  13. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  14. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 150
  15. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  16. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  17. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  18. 2019With Malice Towards None: Assessing Uncertainty via Equalized Coveragecited by 43
  19. 2020PAC Confidence Predictions for Deep Neural Network Classifierscited by 21
  20. 2021RomeBERT: Robust Training of Multi-Exit BERTcited by 19
  21. 2020Knowing what you know: valid and validated confidence sets in multiclass and multilabel predictioncited by 9
  22. 2017A Deep Network with Visual Text Composition Behaviorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 19; 2024: 22; 2025: 7; 2026: 9.

73 papers.

Sort:

By area:

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  4. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  5. 2022Confident Adaptive Language Modelingcited by 297
  6. 2022Conformal Risk Controlcited by 290
  7. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  8. 2023Conformal Language Modelingcited by 163
  9. 2022FlexiViT: One Model for All Patch Sizescited by 151
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  12. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
38 more
  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  3. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  4. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  5. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  6. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  7. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  8. 2021A Survey on Green Deep Learningcited by 52
  9. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  10. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  11. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  12. 2024A Survey on Transformer Compressioncited by 44
  13. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  14. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  15. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  16. 2025Efficient Diffusion Models: A Surveycited by 35
  17. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  18. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  19. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  20. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  21. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  22. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
  23. 2023A Study on the Calibration of In-context Learningcited by 21
  24. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  25. 2023F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Taskscited by 18
  26. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
  27. 2024Presto! Distilling Steps and Layers for Accelerating Music Generationcited by 16
  28. 2022Recommendation Systems with Distribution-Free Reliability Guaranteescited by 13
  29. 2023On the Expected Size of Conformal Prediction Setscited by 13
  30. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inferencecited by 13
  31. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  32. 2024A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Modelscited by 13
  33. 2024CEEBERT: Cross-Domain Inference in Early Exit BERTcited by 12
  34. 2024Early Time Classification with Accumulated Accuracy Gap Controlcited by 11
  35. 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferencecited by 9
  36. 2025Language Models Can Predict Their Own Behaviorcited by 8
  37. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  38. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8

50 most cited of 73 All 73 (JSON)