# PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference

arXiv: https://arxiv.org/abs/2405.14430 · HTML: https://mlsanity.com/paper/2405.14430/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 23 May 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models](https://mlsanity.com/paper/2402.19481/) (2024)
- [TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models](https://mlsanity.com/paper/2102.07988/) (2021)
- [USP: A Unified Sequence Parallelism Approach for Long Context Generative AI](https://mlsanity.com/paper/2405.07719/) (2024)

## Built on by (most cited first, 28 of 28)

- [Training-free and Adaptive Sparse Attention for Efficient Long Video Generation](https://mlsanity.com/paper/2502.21079/) (2025)
- [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025)
- [xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism](https://mlsanity.com/paper/2411.01738/) (2024)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation](https://mlsanity.com/paper/2511.07399/) (2025)
- [QVGen: Pushing the Limit of Quantized Video Generative Models](https://mlsanity.com/paper/2505.11497/) (2025)
- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)
- [SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing](https://mlsanity.com/paper/2603.08982/) (2026)
- [SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation](https://mlsanity.com/paper/2605.30116/) (2026)
- [db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism](https://mlsanity.com/paper/2511.23113/) (2025)
- [M*: A Modular, Extensible, Serving System for Multimodal Models](https://mlsanity.com/paper/2606.12688/) (2026)
- [Communication-Efficient Diffusion Denoising Parallelization via Reuse-then-Predict Mechanism](https://mlsanity.com/paper/2505.14741/) (2025)
- [CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems](https://mlsanity.com/paper/2508.11287/) (2025)
- [SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs](https://mlsanity.com/paper/2601.20273/) (2026)
- [PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models](https://mlsanity.com/paper/2605.09503/) (2026)
- [Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion](https://mlsanity.com/paper/2605.18346/) (2026)
- [EcoVideo: Entropy-Orchestrated Video Generation Paradigm in Cloud-Edge Dynamics](https://mlsanity.com/paper/2606.30557/) (2026)
- [VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention](https://mlsanity.com/paper/2609.15810/) (2026)
- [Minute-Long Videos with Dual Parallelisms](https://mlsanity.com/paper/2505.21070/) (2025)
- [GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving](https://mlsanity.com/paper/2606.13501/) (2026)
- [ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate](https://mlsanity.com/paper/2607.28144/) (2026)
- [Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation](https://mlsanity.com/paper/2609.20744/) (2026)
- [Accelerating Diffusion via Hybrid Data-Pipeline Parallelism Based on Conditional Guidance Scheduling](https://mlsanity.com/paper/2602.21760/) (2026)
- [ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference](https://mlsanity.com/paper/2605.11335/) (2026)
- [Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPU](https://mlsanity.com/paper/2606.05981/) (2026)
- [AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformers](https://mlsanity.com/paper/2606.17566/) (2026)
- [SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTs](https://mlsanity.com/paper/2609.32197/) (2026)
- [In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion](https://mlsanity.com/paper/2609.32540/) (2026)

## Abstract (arXiv)

This paper presents PipeFusion, an innovative parallel methodology to tackle the high latency issues associated with generating high-resolution images using diffusion transformers (DiTs) models. PipeFusion partitions images into patches and the model layers across multiple GPUs. It employs a patch-level pipeline parallel strategy to orchestrate communication and computation efficiently. By capitalizing on the high similarity between inputs from successive diffusion steps, PipeFusion reuses one-step stale feature maps to provide context for the current pipeline step. This approach notably reduces communication costs compared to existing DiTs inference parallelism, including tensor parallel, sequence parallel and DistriFusion. PipeFusion enhances memory efficiency through parameter distribution across devices, ideal for large DiTs like Flux.1. Experimental results demonstrate that PipeFusion achieves state-of-the-art performance on 8×L40 PCIe GPUs for Pixart, Stable-Diffusion 3, and Flux.1 models. Our source code is available at https://github.com/xdit-project/xDiT.