mlsanityData to 14 Oct 2026

Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism

arXiv 2511.06247 ↗First posted 9 Nov 2025cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing state-of-the-art systems tune the sequence parallelism (SP) allocation. However, current dynamic SP allocation lacks flexibility to (1) support stage-specific parallelism requirements in LLM inference, (2) mitigate the global latency degradation from excessive SP allocation, and (3) exploit resource fragments arising from SP size variation. To tackle this problem, we propose Chunkwise Dynamic Sequence Parallelism (CDSP), a fine-grained parallelism strategy that assigns SP sizes across textitintra-request token segments. Based on CDSP, we build Tetris, an LLM serving system that (1) efficiently integrates CDSP into disaggregated cluster to satisfy parallelism heterogeneity, (2) dynamically regulates SP size expansion based on real-time load conditions, and (3) adaptively explores chunking plans to utilize fragmented resources while meeting per-request demands. Compared with state-of-the-art systems, Tetris achieves up to 4.35× lower time-to-first-token (TTFT) under max sustainable loads, reduces median time-between-tokens (TBT) by up to 40.1%, and increases the max request capacity by up to 45%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 35 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

18 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  10. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  11. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  12. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
3 more
  1. 2025Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compilercited by 25
  2. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  3. 2024LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelismcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  2. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  3. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
  4. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  5. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  6. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  7. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
  8. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving