Data to 7 Oct 2026

The Right Tool for the Job: Matching Model and Instance Complexities

arXiv 2004.07453 ↗First posted 16 Apr 2020cs.CL cs.LGcited by 125

Not on the map. Why?

Cited by 32 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. To better respect a given inference budget, we propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" from neural network calculations for simple instances, and late (and accurate) exit for hard instances. To achieve this, we add classifiers to different layers of BERT and use their calibrated confidence scores to make early exit decisions. We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks. Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy. Our method also requires almost no additional training resources (in either time or parameters) compared to the baseline BERT model. Finally, our method alleviates the need for costly retraining of multiple models at different levels of efficiency; we allow users to control the inference speed/accuracy tradeoff using a single trained model, by setting a single variable at inference time. We publicly release our code.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 20; 2023: 34; 2024: 24; 2025: 7; 2026: 16.

125 papers.

Sort:

By area:

2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
2021Dynamic Neural Networks: A Surveycited by 405
2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
2022Confident Adaptive Language Modelingcited by 297
2023Think before you speak: Training Language Models With Pause Tokenscited by 254
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2021Deep Learning Through the Lens of Example Difficultycited by 139
2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89

38 more

2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
2023Efficient Benchmarking of Language Modelscited by 71
2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 54
2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
2021A Survey on Green Deep Learningcited by 52
2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
2022Generative Knowledge Graph Construction: A Reviewcited by 49
2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
2024A Survey on Transformer Compressioncited by 44
2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 28
2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
2021End-to-End Self-Debiasing Framework for Robust NLU Trainingcited by 24
2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
2023Adaptive Computation with Elastic Input Sequencecited by 22
2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
2021Early Exiting with Ensemble Internal Classifierscited by 19
2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
2023When Do Program-of-Thoughts Work for Reasoning?cited by 19
2023Learning to Skip for Language Modelingcited by 19
2024Online Cascade Learning for Efficient Inference over Streamscited by 19

50 most cited of 125 All 125 (JSON)