mlsanityData to 14 Oct 2026

TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling

arXiv 2510.02758 ↗First posted 3 Oct 2025cs.LGcited by 10

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Real-time LLM interactions demand streamed token generations, where text tokens are progressively generated and delivered to users while balancing two objectives: responsiveness (i.e., low time-to-first-token) and steady generation (i.e.,required time-between-tokens). Standard LLM serving systems suffer from the inflexibility caused by non-preemptive request scheduling and reactive memory management, leading to poor resource utilization and low request processing parallelism under request bursts. Therefore, we present TokenFlow, a novel LLM serving system with enhanced text streaming performance via preemptive request scheduling and proactive key-value (KV) cache management. TokenFlow dynamically prioritizes requests based on real-time token buffer occupancy and token consumption rate, while actively transferring KV cache between GPU and CPU memory in the background and overlapping I/O with computation to minimize request preemption overhead. Extensive experiments on Llama3-8B and Qwen2.5-32B across multiple GPUs (RTX 4090, A6000, H200) demonstrate that TokenFlow achieves up to 82.5% higher effective throughput (accounting for actual user consumption) while reducing P99 TTFT by up to 80.2%, without degrading overall token throughput.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 70 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

47 of 70 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  2. 2025Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving25 Apr
  3. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  4. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving19 Jun
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  6. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  7. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec
  12. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
23 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 539
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  7. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  8. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  9. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  10. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  11. 2023Stateful Large Language Model Serving with Pensievecited by 61
  12. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
  13. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  14. 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 45
  15. 2024Understanding User Experience in Large Language Model Interactionscited by 44
  16. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  17. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  18. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  19. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 18
  20. 2024Task Supportive and Personalized Human-Large Language Model Interaction: A User Studycited by 17
  21. 2025Pre3: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generationcited by 11
  22. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  23. 2025Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Corescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoningcited by 4
  2. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inferencecited by 4
  3. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  4. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  5. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  6. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Accelerationcited by 2
  7. 2026LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inferencecited by 1
  8. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  9. 2026LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Modelscited by 0
  10. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
  2. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
  3. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
  4. 2026Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
  5. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Serving
  6. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
  7. 2026AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications
  8. 2026StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving