mlsanityData to 7 Oct 2026

UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing

arXiv 2605.18796 ↗First posted 11 May 2026cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per-workload threshold tuning. We present UCCI, a calibration-first router that maps token-level margin uncertainty to a per-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization. Under three explicit assumptions, threshold policies on the calibrated score are cost-optimal, and isotonic calibration achieves O(n^{-1/3}) sample complexity for expected calibration error (ECE). On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% (95% CI: [27%, 35%]) at micro-F1 = 0.91 while reducing ECE from 0.12 to 0.03. At the same operating point, UCCI beats entropy thresholding, split-conformal routing, and a FrugalGPT-style learned threshold. All cascade results use end-to-end routing on actual model outputs and measured H100 latency, not simulated routing from global accuracies or nominal API prices.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

18 of 23 references.

References · 18

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2017Selective Classification for Deep Neural Networkscited by 799
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  6. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  10. 2022Confident Adaptive Language Modelingcited by 297
  11. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
  12. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
6 more
  1. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  2. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  3. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  4. 2024Retrieval Augmented Generation for Domain-specific Question Answeringcited by 22
  5. 2025Cost-Saving LLM Cascades with Early Abstentioncited by 14
  6. 2026PromptPort: A Reliability Layer for Cross-Model Structured Extractioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.