mlsanityData to 7 Oct 2026

Context Parallelism for Scalable Million-Token Inference

arXiv 2411.01783 ↗First posted 4 Nov 2024cs.DC cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

We present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method achieves 1M context prefill with Llama3 405B model in 77s (93% parallelization efficiency, 63% FLOPS utilization) and 128K context prefill in 3.8s. We develop two lossless exact ring attention variants: pass-KV and pass-Q to cover a wide range of use cases with the state-of-the-art performance: full prefill, persistent KV prefill and decode. Benchmarks on H100 GPU hosts inter-connected with RDMA and TCP both show similar scalability for long-context prefill, demonstrating that our method scales well using common commercial data center with medium-to-low inter-host bandwidth.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
27 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2022Efficiently Scaling Transformer Inferencecited by 529
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  13. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  14. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  15. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  16. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  17. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  18. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  19. 2018Online normalizer calculation for softmaxcited by 154
  20. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  21. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  22. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  23. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 58
  24. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  25. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 28
  26. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  27. 2024KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  2. 2026Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstructioncited by 25
  3. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 14
  4. 2025Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulationcited by 8
  5. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  6. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decodingcited by 7
  7. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  8. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  9. 2025Tensor-Parallelism with Partially Synchronized Activationscited by 4
  10. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
  11. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  12. 2025Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelismcited by 1
13 more
  1. 2025Design Space Exploration of DMA based Finer-Grain Compute Communication Overlapcited by 1
  2. 2025Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Localitycited by 1
  3. 2026Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Modelscited by 1
  4. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  5. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  6. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 1
  7. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottleneckscited by 0
  8. 2026NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learningcited by 0
  9. 2026The qs Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inferencecited by 0
  10. 2026Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Servingcited by 0
  11. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0
  12. 2026KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decodingcited by 0
  13. 2026SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Servingcited by 0