# RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

arXiv: https://arxiv.org/abs/2610.18143 · HTML: https://mlsanity.com/paper/2610.18143/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 21 of 21)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference](https://mlsanity.com/paper/2510.09665/) (2025)
- [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention](https://mlsanity.com/paper/2405.04437/) (2024)
- [Preble: Efficient Distributed Prompt Scheduling for LLM Serving](https://mlsanity.com/paper/2407.00023/) (2024)
- [BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching](https://mlsanity.com/paper/2412.03594/) (2024)
- [Marconi: Prefix Caching for the Era of Hybrid LLMs](https://mlsanity.com/paper/2411.19379/) (2024)
- [LLM Query Scheduling with Prefix Reuse and Latency Constraints](https://mlsanity.com/paper/2502.04677/) (2025)

## Abstract (arXiv)

Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Schedulers assume that a matched prefix is ready, which holds for full attention but fails for hybrid LLMs. There, a branch (the consumer) may need recurrent state that another branch (the producer) is still computing in the same round. The native scheduler stops its waiting-queue scan at this blocked consumer, and the resulting head-of-line blocking creates execution bubbles: idle batch slots that independent work could fill. We introduce RadixFlow, which decouples prefix matching from state readiness. Its state-ready traversal turns a global scan barrier into a local wait: only the consumer is blocked until the next round, while feasible independent branches keep filling the batch under the native state check. On five hybrid models, RadixFlow lowers mean latency by 3% to 24% and worker rounds by 11% to 44% at almost unchanged reuse. On the larger Qwen3.5 models, ablations show that most of the gain comes from traversal, which adds little on a full-attention control. Heavier rounds limit how much of this saving reaches latency.