mlsanityData to 14 Oct 2026

Language Model Cascades: Token-level uncertainty and beyond

arXiv 2404.10136 ↗First posted 15 Apr 2024cs.CL cs.AI cs.LGcited by 121

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in language models (LMs) have led to significant improvements in quality on complex NLP tasks, but at the expense of increased inference costs. Cascading offers a simple strategy to achieve more favorable cost-quality tradeoffs: here, a small model is invoked for most "easy" instances, while a few "hard" instances are deferred to the large model. While the principles underpinning cascading are well-studied for classification tasks - with deferral based on predicted class uncertainty favored theoretically and practically - a similar understanding is lacking for generative LM tasks. In this work, we initiate a systematic study of deferral rules for LM cascades. We begin by examining the natural extension of predicted class uncertainty to generative LM tasks, namely, the predicted sequence uncertainty. We show that this measure suffers from the length bias problem, either over- or under-emphasizing outputs based on their lengths. This is because LMs produce a sequence of uncertainty values, one for each output token; and moreover, the number of output tokens is variable across examples. To mitigate this issue, we propose to exploit the richer token-level uncertainty information implicit in generative LMs. We argue that naive predicted sequence uncertainty corresponds to a simple aggregation of these uncertainties. By contrast, we show that incorporating token-level uncertainty through learned post-hoc deferral rules can significantly outperform such simple aggregation strategies, via experiments on a range of natural language benchmarks with FLAN-T5 models. We further show that incorporating embeddings from the smaller model and intermediate layers of the larger model can give an additional boost in the overall cost-quality tradeoff.

Key facts. Cited by 121 papers in the map (data through 14 Oct 2026). Builds on 61 of its 85 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

61 of 85 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
38 more
  1. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2018Hierarchical Neural Story Generationcited by 1,193
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  14. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  15. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  16. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  17. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  18. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  19. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  20. 2022Confident Adaptive Language Modelingcited by 310
  21. 2022UL2: Unifying Language Learning Paradigmscited by 295
  22. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  23. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  24. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  25. 2018To Trust Or Not To Trust A Classifiercited by 207
  26. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  27. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  28. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  29. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  30. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  31. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  32. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  33. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  34. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  35. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  36. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  37. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  38. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 48

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 49; 2026: 50.

121 papers.

Sort:

By area:

  1. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  3. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  4. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  5. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  6. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  7. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  8. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
  9. 2025Reasoning Models Better Express Their Confidencecited by 65
  10. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  11. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 39
  12. 2024Faster Cascades via Speculative Decodingcited by 38
38 more
  1. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  2. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  3. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  4. 2025Estimating LLM Uncertainty with Evidencecited by 26
  5. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  6. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 25
  7. 2024A Survey on the Honesty of Large Language Modelscited by 23
  8. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 23
  9. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  10. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  11. 2024Prune 'n Predict: Optimizing LLM Decision-making with Conformal Predictioncited by 18
  12. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  13. 2024On Subjective Uncertainty Quantification and Calibration in Natural Language Generationcited by 16
  14. 2024Agreement-Based Cascading for Efficient Inferencecited by 16
  15. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 16
  16. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  17. 2024Cascade-Aware Training of Language Modelscited by 15
  18. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  19. 2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 14
  20. 2025Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcementcited by 14
  21. 2025Query-Level Uncertainty in Large Language Modelscited by 14
  22. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  23. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 12
  24. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  25. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  26. 2025Rational Tuning of LLM Cascades via Probabilistic Modelingcited by 10
  27. 2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
  28. 2025Don't Throw Away Your Pretrained Modelcited by 10
  29. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  30. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  31. 2025Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detectioncited by 8
  32. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
  33. 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7
  34. 2026Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systemscited by 7
  35. 2025PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databasescited by 6
  36. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  37. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6
  38. 2026AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devicescited by 6

50 most cited of 121 All 121 (JSON)