mlsanityData to 14 Oct 2026

OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration

arXiv 2601.10729 ↗First posted 5 Jan 2026cs.AI cs.LG cs.PFcited by 6

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving long-context LLMs is challenging because request lengths and batch composition vary during token generation, causing the memory footprint to fluctuate significantly at runtime. Offloading KV caches to host memory limits effective memory usage, but existing static and predetermined offloading strategies cannot adapt to the rapidly shifting memory demands of long-context serving. This often leads to excessive CPU-to-GPU KV transfers that translate into latency spikes and frequent SLO violations. To address these challenges, we introduce OrbitFlow, a fine-grained and adaptive KV cache management system that meets latency SLOs in long-context LLM serving. OrbitFlow employs a lightweight ILP solver to decide which layers' KV caches to retain on the GPU for each request, within memory capacity constraints. It continuously refines KV placements based on runtime feedback when the active plan becomes suboptimal during token generation. Under heavy load, OrbitFlow invokes a fallback mechanism to temporarily defer in-flight requests with large memory footprints, preserving overall SLO attainment. Our experiments demonstrate that OrbitFlow improves SLO attainment for TPOT and TBT by up to 66% and 48%, respectively, while reducing the 95th percentile latency by 38% and achieving up to 3.3x higher throughput compared to existing offloading methods.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 53 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

37 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference2 Nov
  2. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  3. 2024Queue management for slo-oriented large language model serving5 Jun
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023Fairness in Serving Large Language Models31 Dec

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
19 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  9. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  10. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  11. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  12. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  13. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  14. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  15. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 48
  16. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  17. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  18. 2025Memory Offloading for Large Language Model Inference with Latency SLO Guaranteescited by 3
  19. 2024Learned Best-Effort LLM Servingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 3
  2. 2026StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Servingcited by 2
  3. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  4. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1
  5. 2026SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Servingcited by 1
  6. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  2. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  3. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  4. 2026Unified KV Pooling to Accelerate Long-Context LLM Serving
  5. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  6. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving
  7. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  8. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading