# DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers

arXiv: https://arxiv.org/abs/2403.10266 · HTML: https://mlsanity.com/paper/2403.10266/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2024.

## Built on by (most cited first, 15 of 15)

- [Real-Time Video Generation with Pyramid Attention Broadcast](https://mlsanity.com/paper/2408.12588/) (2024)
- [GameGen-X: Interactive Open-world Game Video Generation](https://mlsanity.com/paper/2411.00769/) (2024)
- [FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality](https://mlsanity.com/paper/2410.19355/) (2024)
- [Adaptive Caching for Faster Video Generation with Diffusion Transformers](https://mlsanity.com/paper/2411.02397/) (2024)
- [LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation](https://mlsanity.com/paper/2605.18739/) (2026)
- [BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching](https://mlsanity.com/paper/2509.13789/) (2025)
- [LeMiCa: Lexicographic Minimax Path Caching for Efficient Diffusion-Based Video Generation](https://mlsanity.com/paper/2511.00090/) (2025)
- [MeanCache: From Instantaneous to Average Velocity for Accelerating Flow Matching Inference](https://mlsanity.com/paper/2601.19961/) (2026)
- [PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generation](https://mlsanity.com/paper/2603.00976/) (2026)
- [SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation](https://mlsanity.com/paper/2505.19151/) (2025)
- [HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing](https://mlsanity.com/paper/2603.07236/) (2026)
- [PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling](https://mlsanity.com/paper/2511.12056/) (2025)
- [OTCache: Optimal Transport for Geometry-Aware Caching in Diffusion Models](https://mlsanity.com/paper/2606.31026/) (2026)
- [Training Video Foundation Models with NVIDIA NeMo](https://mlsanity.com/paper/2503.12964/) (2025)
- [DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse](https://mlsanity.com/paper/2607.15650/) (2026)

## Abstract (arXiv)

Scaling multi-dimensional transformers to long sequences is indispensable across various domains. However, the challenges of large memory requirements and slow speeds of such sequences necessitate sequence parallelism. All existing approaches fall under the category of embedded sequence parallelism, which are limited to shard along a single sequence dimension, thereby introducing significant communication overhead. However, the nature of multi-dimensional transformers involves independent calculations across multiple sequence dimensions. To this end, we propose Dynamic Sequence Parallelism (DSP) as a novel abstraction of sequence parallelism. DSP dynamically switches the parallel dimension among all sequences according to the computation stage with efficient resharding strategy. DSP offers significant reductions in communication costs, adaptability across modules, and ease of implementation with minimal constraints. Experimental evaluations demonstrate DSP's superiority over state-of-the-art embedded sequence parallelism methods by remarkable throughput improvements ranging from 32.2% to 10x, with less than 25% communication volume.