mlsanityData to 14 Oct 2026

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

arXiv 2610.18143 ↗First posted 13 Oct 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Schedulers assume that a matched prefix is ready, which holds for full attention but fails for hybrid LLMs. There, a branch (the consumer) may need recurrent state that another branch (the producer) is still computing in the same round. The native scheduler stops its waiting-queue scan at this blocked consumer, and the resulting head-of-line blocking creates execution bubbles: idle batch slots that independent work could fill. We introduce RadixFlow, which decouples prefix matching from state readiness. Its state-ready traversal turns a global scan barrier into a local wait: only the consumer is blocked until the next round, while feasible independent branches keep filling the batch under the native state check. On five hybrid models, RadixFlow lowers mean latency by 3% to 24% and worker rounds by 11% to 44% at almost unchanged reuse. On the larger Qwen3.5 models, ablations show that most of the gain comes from traversal, which adds little on a full-attention control. Heavier rounds limit how much of this saving reaches latency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 34 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

21 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  2. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 17

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  12. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
5 more
  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  3. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  4. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  5. 2025LLM Query Scheduling with Prefix Reuse and Latency Constraintscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
  2. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  3. 2026PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving
  4. 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
  5. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  6. 2024A System for Microserving of LLMs
  7. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems
  8. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving