mlsanityData to 7 Oct 2026

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

arXiv 2404.09526 ↗First posted 15 Apr 2024cs.DC cs.LGcited by 110

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance between different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the maximum throughput by up to 3.85× compared to the chunked prefill and 5.81× compared to the prefill-decoding disaggregation.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 54 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

30 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
17 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  5. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  6. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  7. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  8. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  9. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  10. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  11. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  12. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  13. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  14. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  15. 2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 24
  16. 2022EasyScale: Accuracy-consistent Elastic Training for Deep Learningcited by 3
  17. 2021Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Serverscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 54; 2026: 39.

110 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  3. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  4. 2024Fast On-device LLM Inference with NPUscited by 58
  5. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  6. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  8. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  9. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
  10. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  11. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  12. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
38 more
  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  2. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  3. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  4. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25
  5. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  6. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  7. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  8. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  9. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  10. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  11. 2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13
  12. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  13. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
  14. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 12
  15. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  16. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  17. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  18. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  19. 2025On Evaluating Performance of LLM Inference Serving Systemscited by 9
  20. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  21. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decodingcited by 7
  22. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  23. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 7
  24. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  25. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  26. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  27. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 5
  28. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  29. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5
  30. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  31. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 5
  32. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  33. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  34. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
  35. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  36. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Servingcited by 4
  37. 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMscited by 4
  38. 2025EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Modelscited by 4

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  2. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  4. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  5. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  6. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
  7. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  8. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption