mlsanityData to 14 Oct 2026

Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding

arXiv 2507.07120 ↗First posted 7 Jul 2025cs.DC cs.AIcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

As LLMs scale to multi-million-token KV histories, real-time autoregressive decoding under tight Token-to-Token Latency (TTL) constraints faces growing pressure. Two core bottlenecks dominate: accessing Feed-Forward Network (FFN) weights and reading long KV caches. While Tensor Parallelism (TP) helps mitigate the cost of FFN weight reads, it does not scale well for attention. When TP width exceeds the number of KV heads, it leads to inefficient KV duplication, limits parallelism, and constrains batch size. Simultaneously, DRAM reads for long KV histories scale linearly with batch size, further capping efficiency. We introduce Helix Parallelism, a hybrid execution strategy that applies KV parallelism during attention to shard KV caches across GPUs, then reuses the same GPUs for TP in dense LLMs or TPxExpert Parallel (EP) in MoEs during FFN computation. To preserve exact attention behavior, Helix includes a lightweight communication step. To minimize the exposed communication cost, we introduce Helix HOP-B. Helix HOP-B effectively minimizes communication overhead through batchwise overlap, preserving low TTL while improving GPU efficiency. Compared to conventional parallelism approaches, Helix reduces TTL by up to 1.5x at fixed batch sizes and supports up to 32x larger batches under the same latency budget for DeepSeek-R1, pushing forward the throughput-latency Pareto on Blackwell and making real-time inference with ultra-long-sequence practical.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 13 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

6 of 13 references.

References · 6

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  5. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 29
  6. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  2. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  3. 2025Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inferencecited by 0
  4. 2026The qs Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inferencecited by 0
  5. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0
  6. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inferencecited by 0
  7. 2026WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decoding
  2. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  4. 2025Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding
  5. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
  6. 2025BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  8. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration