# Cascadia: An Efficient Cascade Serving System for Large Language Models arXiv: https://arxiv.org/abs/2506.04203 · HTML: https://mlsanity.com/paper/2506.04203/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. - Builds on 38 of its 49 references that are in the map. - First posted on arXiv 4 Jun 2025. ## Placed in - Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/) ## Builds on (most cited first, 30 of 38) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024) - [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023) - [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023) - [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024) - [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024) - [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017) - [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023) - [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024) - [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023) - [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023) - [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024) - [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023) - [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023) - [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023) - [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024) - [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023) - [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023) - [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024) - [Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism](https://mlsanity.com/paper/2211.13878/) (2022) - [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024) - [Faster Cascades via Speculative Decoding](https://mlsanity.com/paper/2405.19261/) (2024) - [HexGen: Generative Inference of Large Language Model over Heterogeneous Environment](https://mlsanity.com/paper/2311.11514/) (2023) - [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://mlsanity.com/paper/2502.09334/) (2025) - [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs](https://mlsanity.com/paper/2502.00722/) (2025) ## Built on by (most cited first, 5 of 5) - [Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoning](https://mlsanity.com/paper/2508.05282/) (2025) - [Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods](https://mlsanity.com/paper/2605.05227/) (2026) - [Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics](https://mlsanity.com/paper/2604.07144/) (2026) - [Learning to Solve Complex Problems via Dataset Decomposition](https://mlsanity.com/paper/2602.20296/) (2026) - [HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling](https://mlsanity.com/paper/2605.16637/) (2026) ## Abstract (arXiv) Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency-quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy. Motivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4× (2.3× on average) tighter latency SLOs and up to 5× (2.4× on average) higher throughput while maintaining target answer quality.