UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
Not on the map. Why?
Abstract (arXiv)
LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per-workload threshold tuning. We present UCCI, a calibration-first router that maps token-level margin uncertainty to a per-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization. Under three explicit assumptions, threshold policies on the calibrated score are cost-optimal, and isotonic calibration achieves O(n^{-1/3}) sample complexity for expected calibration error (ECE). On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% (95% CI: [27%, 35%]) at micro-F1 = 0.91 while reducing ECE from 0.12 to 0.03. At the same operating point, UCCI beats entropy thresholding, split-conformal routing, and a FrugalGPT-style learned threshold. All cascade results use end-to-end routing on actual model outputs and measured H100 latency, not simulated routing from global accuracies or nominal API prices.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
18 of 23 references.
References · 18
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
- 2022Confident Adaptive Language Modelingcited by 297
- 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
6 more
- 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
- 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
- 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
- 2024Retrieval Augmented Generation for Domain-specific Question Answeringcited by 22
- 2025Cost-Saving LLM Cascades with Early Abstentioncited by 14
- 2026PromptPort: A Reliability Layer for Cross-Model Structured Extractioncited by 3
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systemscited by 1
- 2026The Handoff Tax: Continuing Non-Native Trajectories in LLM Agentscited by 1
- 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprisecited by 1
- 2026Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflowscited by 0
- 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise24 Sep 2026
- 2026The Handoff Tax: Continuing Non-Native Trajectories in LLM Agents25 Aug 2026
- 2026Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systems13 Jul 2026
- 2026Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows12 Jul 2026