# DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

arXiv: https://arxiv.org/abs/2402.19481 · HTML: https://mlsanity.com/paper/2402.19481/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 64 of its 79 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 64)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)

## Built on by (most cited first, 30 of 98)

- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models](https://mlsanity.com/paper/2410.10733/) (2024)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://mlsanity.com/paper/2411.05007/) (2024)
- [Real-Time Video Generation with Pyramid Attention Broadcast](https://mlsanity.com/paper/2408.12588/) (2024)
- [Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching](https://mlsanity.com/paper/2406.01733/) (2024)
- [XAttention: Block Sparse Attention with Antidiagonal Scoring](https://mlsanity.com/paper/2503.16428/) (2025)
- [Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation](https://mlsanity.com/paper/2505.18875/) (2025)
- [Training-free and Adaptive Sparse Attention for Efficient Long Video Generation](https://mlsanity.com/paper/2502.21079/) (2025)
- [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025)
- [xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism](https://mlsanity.com/paper/2411.01738/) (2024)
- [LinFusion: 1 GPU, 1 Minute, 16K Image](https://mlsanity.com/paper/2409.02097/) (2024)
- [VMoBA: Mixture-of-Block Attention for Video Diffusion Models](https://mlsanity.com/paper/2506.23858/) (2025)
- [Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.15734/) (2025)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient](https://mlsanity.com/paper/2411.17787/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Accelerated Diffusion Models via Speculative Sampling](https://mlsanity.com/paper/2501.05370/) (2025)
- [Video-Infinity: Distributed Long Video Generation](https://mlsanity.com/paper/2406.16260/) (2024)
- [StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation](https://mlsanity.com/paper/2511.07399/) (2025)
- [PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference](https://mlsanity.com/paper/2405.14430/) (2024)
- [CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up](https://mlsanity.com/paper/2412.16112/) (2024)
- [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024)
- [Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile](https://mlsanity.com/paper/2502.06155/) (2025)
- [AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising](https://mlsanity.com/paper/2406.06911/) (2024)
- [DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance](https://mlsanity.com/paper/2505.14708/) (2025)
- [A Survey on Diffusion Models for Recommender Systems](https://mlsanity.com/paper/2409.05033/) (2024)
- [FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion](https://mlsanity.com/paper/2506.04648/) (2025)
- [CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models](https://mlsanity.com/paper/2502.00433/) (2025)
- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)

## Abstract (arXiv)

Diffusion models have achieved great success in synthesizing high-quality images. However, generating high-resolution images with diffusion models is still challenging due to the enormous computational costs, resulting in a prohibitive latency for interactive applications. In this paper, we propose DistriFusion to tackle this problem by leveraging parallelism across multiple GPUs. Our method splits the model input into multiple patches and assigns each patch to a GPU. However, naively implementing such an algorithm breaks the interaction between patches and loses fidelity, while incorporating such an interaction will incur tremendous communication overhead. To overcome this dilemma, we observe the high similarity between the input from adjacent diffusion steps and propose displaced patch parallelism, which takes advantage of the sequential nature of the diffusion process by reusing the pre-computed feature maps from the previous timestep to provide context for the current step. Therefore, our method supports asynchronous communication, which can be pipelined by computation. Extensive experiments show that our method can be applied to recent Stable Diffusion XL with no quality degradation and achieve up to a 6.1× speedup on eight NVIDIA A100s compared to one. Our code is publicly available at https://github.com/mit-han-lab/distrifuser.