mlsanityData to 14 Oct 2026

Cascade-Aware Training of Language Models

arXiv 2406.00060 ↗First posted 29 May 2024cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Reducing serving cost and latency is a fundamental concern for the deployment of language models (LMs) in business applications. To address this, cascades of LMs offer an effective solution that conditionally employ smaller models for simpler queries. Cascaded systems are typically built with independently trained models, neglecting the advantages of considering inference-time interactions of the cascaded LMs during training. In this paper, we present cascade-aware training(CAT), an approach to optimizing the overall quality-cost performance tradeoff of a cascade of LMs. We achieve inference-time benefits by training the small LM with awareness of its place in a cascade and downstream capabilities. We demonstrate the value of the proposed method with over 60 LM tasks of the SuperGLUE, WMT22, and FLAN2021 datasets.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 51 of its 65 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

51 of 65 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2023PaLM 2 Technical Reportcited by 1,162
38 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  2. 2016Sequence-Level Knowledge Distillationcited by 801
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  4. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  5. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  6. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  7. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  8. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  9. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  10. 2022Confident Adaptive Language Modelingcited by 310
  11. 2022Matryoshka Representation Learningcited by 282
  12. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  13. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  14. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  15. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  16. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  17. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  18. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  19. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  20. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
  21. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  22. 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
  23. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  24. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
  25. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  26. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 48
  27. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  28. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  29. 2022Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalizationcited by 33
  30. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  31. 2023Tryage: Real-time, intelligent Routing of User Prompts to Large Language Modelscited by 30
  32. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
  33. 2023Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNNcited by 15
  34. 2022TangoBERT: Reducing Inference Cost by using Cascaded Architecturecited by 13
  35. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  36. 2022BabyBear: Cheap inference triage for expensive language modelscited by 10
  37. 2018Approximation Algorithms for Cascading Prediction Modelscited by 9
  38. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  2. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  3. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  4. 2025Rational Tuning of LLM Cascades via Probabilistic Modelingcited by 10
  5. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  6. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  7. 2025Economic Evaluation of LLMscited by 5
  8. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 5
  9. 2025Learning to Partially Defer for Sequencescited by 4
  10. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  11. 2026Streaming Model Cascades for Semantic SQLcited by 3
  12. 2025Bi-directional Model Cascading with Proxy Confidencecited by 2
3 more
  1. 2026Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMscited by 2
  2. 2026LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Losscited by 0
  3. 2026AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Searchcited by 0