mlsanityData to 7 Oct 2026

Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems

arXiv 2210.05528 ↗First posted 11 Oct 2022cs.CL cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

Do all instances need inference through the big models for a correct prediction? Perhaps not; some instances are easy and can be answered correctly by even small capacity models. This provides opportunities for improving the computational efficiency of systems. In this work, we present an explorative study on 'model cascading', a simple technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions. Through comprehensive experiments in multiple task settings that differ in the number of models available for cascading (K value), we show that cascading improves both the computational efficiency and the prediction accuracy. For instance, in K=3 setting, cascading saves up to 88.93% computation cost and consistently achieves superior prediction accuracy with an improvement of up to 2.18%. We also study the impact of introducing additional models in the cascade and show that it further increases the efficiency improvements. Finally, we hope that our work will facilitate development of efficient NLP systems making their widespread adoption in real-world applications possible.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

42 of 60 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
30 more
  1. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  7. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  8. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  12. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  13. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  14. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  15. 2019Structured Pruning of Large Language Modelscited by 201
  16. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  17. 2018Dialogue Natural Language Inferencecited by 135
  18. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  19. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  20. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  21. 2021Towards Zero-Label Language Learningcited by 76
  22. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  23. 2019Unsupervised Question Answering by Cloze Translationcited by 59
  24. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  25. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  26. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  27. 2020The Cascade Transformer: an Application for Efficient Answer Sentence Selectioncited by 20
  28. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  29. 2022ILDAE: Instance-Level Difficulty Analysis of Evaluation Datacited by 17
  30. 2021Unsupervised Natural Language Inference Using PHL Triplet Generationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 10; 2025: 13; 2026: 9.

43 papers.

Sort:

By area:

  1. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  2. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  3. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  4. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  5. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  6. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  7. 2023Selectively Answering Ambiguous Questionscited by 63
  8. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  9. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  10. 2024Faster Cascades via Speculative Decodingcited by 38
  11. 2023Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QAcited by 20
  12. 2024Online Cascade Learning for Efficient Inference over Streamscited by 19
31 more
  1. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  2. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  3. 2024Cascade-Aware Training of Language Modelscited by 15
  4. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  5. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  6. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  7. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  8. 2023Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settingscited by 5
  9. 2024DynamicNER: A Dynamic, Multilingual, and Fine-Grained Dataset for LLM-based Named Entity Recognitioncited by 5
  10. 2023Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?cited by 4
  11. 2025Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferralcited by 4
  12. 2025MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platformcited by 3
  13. 2025Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensemblingcited by 3
  14. 2026Select to Think: Unlocking SLM Potential with Local Sufficiencycited by 3
  15. 2024Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensemblingcited by 2
  16. 2025Bi-directional Model Cascading with Proxy Confidencecited by 2
  17. 2025Routesplain: Towards Faithful and Intervenable Routing for Software-related Taskscited by 2
  18. 2026RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimizationcited by 2
  19. 2023A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attributioncited by 1
  20. 2025M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inferencecited by 1
  21. 2025Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferralcited by 1
  22. 2026Computational Arbitrage in AI Model Marketscited by 1
  23. 2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
  24. 2023Making Pre-trained Language Models both Task-solvers and Self-calibratorscited by 0
  25. 2025KiC: Keyword-inspired Cascade for Cost-Efficient Text Generation with LLMscited by 0
  26. 2025T-TAMER: Provably Taming Trade-offs in ML Servingcited by 0
  27. 2026LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Losscited by 0
  28. 2026When Efficiency Backfires: Cascading LLMs Trigger Cascade Failure under Adversarial Attackcited by 0
  29. 2026How Often Should a Recommender Call an LLM? Value-Weighted Routing, Monitoring, and Seasonal Robustnesscited by 0
  30. 2026MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devicescited by 0
  31. 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklistcited by 0