# SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTs

arXiv: https://arxiv.org/abs/2609.32197 · HTML: https://mlsanity.com/paper/2609.32197/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- Diffusion models

## Builds on (most cited first, 24 of 24)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [Tora: Trajectory-oriented Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2407.21705/) (2024)
- [FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality](https://mlsanity.com/paper/2410.19355/) (2024)
- [SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference](https://mlsanity.com/paper/2502.18137/) (2025)
- [Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation](https://mlsanity.com/paper/2505.18875/) (2025)
- [VSA: Faster Video Diffusion with Trainable Sparse Attention](https://mlsanity.com/paper/2505.13389/) (2025)
- [Training-free and Adaptive Sparse Attention for Efficient Long Video Generation](https://mlsanity.com/paper/2502.21079/) (2025)
- [ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://mlsanity.com/paper/2403.17312/) (2024)
- [Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers](https://mlsanity.com/paper/2506.03065/) (2025)
- [PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference](https://mlsanity.com/paper/2405.14430/) (2024)
- [DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training](https://mlsanity.com/paper/2502.07590/) (2025)
- [BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation](https://mlsanity.com/paper/2508.10774/) (2025)
- [DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation](https://mlsanity.com/paper/2605.23445/) (2026)
- [PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference](https://mlsanity.com/paper/2409.15104/) (2024)
- [db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism](https://mlsanity.com/paper/2511.23113/) (2025)
- [Accelerating Parallel Diffusion Model Serving with Residual Compression](https://mlsanity.com/paper/2507.17511/) (2025)
- [SpComm3D: A Framework for Enabling Sparse Communication in 3D Sparse Kernels](https://mlsanity.com/paper/2404.19638/) (2024)
- [FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation](https://mlsanity.com/paper/2607.16190/) (2026)

## Abstract (arXiv)

Diffusion Transformers have become the dominant architecture for video generation. Their substantial computational cost motivates scaling inference across multi-GPU servers, yet efficient scaling remains challenging on commodity GPUs connected via PCIe, whose bandwidth is limited. Although sparse attention substantially reduces computation, its implications for communication remain underexplored. This paper argues that attention sparsity should be treated as a communication primitive. We present SparSP, an efficient sparse sequence parallel communication system that co-designs token distribution, communication routing, and asynchronous execution for sparse video diffusion models. First, Dependency-Aware Placement distributes sequence blocks according to diffusion models' sparse attention patterns. Second, Demand-Directed KV Routing transfers KV blocks directly to requesting GPUs without intermediate relays. Third, a Decoupled Transfer Runtime separates communication from GPU computation to reduce resource contention and maximize effective bandwidth. Our evaluation shows that SparSP improves attention performance by 1.38- 1.5×, achieves an average 1.17× (up to 1.69×) end-to-end speedup across three representative servers and three video diffusion models, and reduces communication volume by 12.54-23.05%. Moreover, we achieve an average 1.53-1.76× bandwidth improvement over NCCL.