mlsanityData to 14 Oct 2026

PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference

arXiv 2405.14430 ↗First posted 23 May 2024cs.CV cs.AI cs.PFcited by 28

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

This paper presents PipeFusion, an innovative parallel methodology to tackle the high latency issues associated with generating high-resolution images using diffusion transformers (DiTs) models. PipeFusion partitions images into patches and the model layers across multiple GPUs. It employs a patch-level pipeline parallel strategy to orchestrate communication and computation efficiently. By capitalizing on the high similarity between inputs from successive diffusion steps, PipeFusion reuses one-step stale feature maps to provide context for the current pipeline step. This approach notably reduces communication costs compared to existing DiTs inference parallelism, including tensor parallel, sequence parallel and DistriFusion. PipeFusion enhances memory efficiency through parameter distribution across devices, ideal for large DiTs like Flux.1. Experimental results demonstrate that PipeFusion achieves state-of-the-art performance on 8×L40 PCIe GPUs for Pixart, Stable-Diffusion 3, and Flux.1 models. Our source code is available at https://github.com/xdit-project/xDiT.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 21 of its 40 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

21 of 40 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  2. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models29 Feb
  3. 2023DeepCache: Accelerating Diffusion Models for Free1 Dec
  4. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  5. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  7. 2022Scalable Diffusion Models with Transformers19 Dec
  8. 2022Classifier-Free Diffusion Guidance26 Jul
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  10. 2020Denoising Diffusion Implicit Models6 Oct

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  6. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  8. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  9. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  10. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  11. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 17.

28 papers.

Sort:

By area:

  1. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  2. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  3. 2024xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelismcited by 56
  4. 2025Efficient Diffusion Models: A Surveycited by 35
  5. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 30
  6. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
  7. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 16
  8. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 10
  9. 2026SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillationcited by 6
  10. 2025db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelismcited by 3
  11. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  12. 2025Communication-Efficient Diffusion Denoising Parallelization via Reuse-then-Predict Mechanismcited by 2
16 more
  1. 2025CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systemscited by 2
  2. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUscited by 2
  3. 2026PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Modelscited by 2
  4. 2026Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusioncited by 2
  5. 2026EcoVideo: Entropy-Orchestrated Video Generation Paradigm in Cloud-Edge Dynamicscited by 2
  6. 2026VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentioncited by 2
  7. 2025Minute-Long Videos with Dual Parallelismscited by 1
  8. 2026GF-DiT: Scheduling Parallelism for Diffusion Transformer Servingcited by 1
  9. 2026ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitratecited by 1
  10. 2026Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationcited by 1
  11. 2026Accelerating Diffusion via Hybrid Data-Pipeline Parallelism Based on Conditional Guidance Schedulingcited by 0
  12. 2026ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inferencecited by 0
  13. 2026Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPUcited by 0
  14. 2026AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformerscited by 0
  15. 2026SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTscited by 0
  16. 2026In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling
  2. 2024Partially Conditioned Patch Parallelism for Accelerated Diffusion Model Inference
  3. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  4. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers
  5. 2025Accelerating Parallel Diffusion Model Serving with Residual Compression
  6. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas
  7. 2026DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction
  8. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer