mlsanityData to 7 Oct 2026

xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism

arXiv 2411.01738 ↗First posted 4 Nov 2024cs.DC cs.AIcited by 55

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models are pivotal for generating high-quality images and videos. Inspired by the success of OpenAI's Sora, the backbone of diffusion models is evolving from U-Net to Transformer, known as Diffusion Transformers (DiTs). However, generating high-quality content necessitates longer sequence lengths, exponentially increasing the computation required for the attention mechanism, and escalating DiTs inference latency. Parallel inference is essential for real-time DiTs deployments, but relying on a single parallel method is impractical due to poor scalability at large scales. This paper introduces xDiT, a comprehensive parallel inference engine for DiTs. After thoroughly investigating existing DiTs parallel approaches, xDiT chooses Sequence Parallel (SP) and PipeFusion, a novel Patch-level Pipeline Parallel method, as intra-image parallel strategies, alongside CFG parallel for inter-image parallelism. xDiT can flexibly combine these parallel approaches in a hybrid manner, offering a robust and scalable solution. Experimental results on two 8xL40 GPUs (PCIe) nodes interconnected by Ethernet and an 8xA100 (NVLink) node showcase xDiT's exceptional scalability across five state-of-the-art DiTs. Notably, we are the first to demonstrate DiTs scalability on Ethernet-connected GPU clusters. xDiT is available at https://github.com/xdit-project/xDiT.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 31 of its 44 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

31 of 44 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Scaling Laws For Diffusion Transformers10 Oct
  2. 2024Scaling Diffusion Transformers to 16 Billion Parameters16 Jul
  3. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching3 Jun
  4. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers3 Jun
  5. 2024PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference23 May
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  7. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models29 Feb
  8. 2023DeepCache: Accelerating Diffusion Models for Free1 Dec
  9. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  11. 2022Scalable Diffusion Models with Transformers19 Dec
  12. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
5 more
  1. 2022Classifier-Free Diffusion Guidance26 Jul
  2. 2022A Study on the Evaluation of Generative Models22 Jun
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2020Denoising Diffusion Implicit Models6 Oct

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  6. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  7. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  9. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  10. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  11. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  12. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 109
2 more
  1. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 58
  2. 2024LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelismcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 23; 2026: 29.

55 papers.

Sort:

By area:

  1. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  2. 2026Infinite Worlds with Versatile Interactionscited by 37
  3. 2025Efficient Diffusion Models: A Surveycited by 35
  4. 2025Training-Free Efficient Video Generation via Dynamic Token Carvingcited by 33
  5. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 32
  6. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 27
  7. 2025Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervalscited by 21
  8. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 20
  9. 2026Where Do the Joules Go? Diagnosing Inference Energy Consumptioncited by 10
  10. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 9
  11. 2025LeMiCa: Lexicographic Minimax Path Caching for Efficient Diffusion-Based Video Generationcited by 9
  12. 2026SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routingcited by 9
38 more
  1. 2025Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulationcited by 8
  2. 2025SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperationcited by 7
  3. 2026MeanCache: From Instantaneous to Average Velocity for Accelerating Flow Matching Inferencecited by 7
  4. 2025SuperGen: An Efficient Ultra-high-resolution Video Generation System with Sketching and Tilingcited by 6
  5. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 6
  6. 2026Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deploymentcited by 6
  7. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  8. 2024FlexCache: Flexible Approximate Cache System for Video Diffusioncited by 4
  9. 2025TetriServe: Efficiently Serving Mixed DiT Workloadscited by 4
  10. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  11. 2024From Noise to Nuance: Advances in Deep Generative Image Modelscited by 3
  12. 2025LazyDrag: Enabling Stable Drag-Based Editing on Multi-Modal Diffusion Transformers via Explicit Correspondencecited by 3
  13. 2025db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelismcited by 3
  14. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  15. 2026An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Modelscited by 3
  16. 2025Communication-Efficient Diffusion Denoising Parallelization via Reuse-then-Predict Mechanismcited by 2
  17. 2025Accelerating Parallel Diffusion Model Serving with Residual Compressioncited by 2
  18. 2026Jano: Adaptive Diffusion Generation with Early-stage Convergence Awarenesscited by 2
  19. 2026HybridStitch: Pixel and Timestep Level Model Stitching for Diffusion Accelerationcited by 2
  20. 2026Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generationcited by 2
  21. 2025Minute-Long Videos with Dual Parallelismscited by 1
  22. 2025CHORDS: Diffusion Sampling Accelerator with Multi-core Hierarchical ODE Solverscited by 1
  23. 2025Spatio-Temporal Parallelism for Diffusion Model Inference on Heterogeneous Multi-GPU Systemscited by 1
  24. 2025PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decouplingcited by 1
  25. 2026Scalable Generative Game Engine: Breaking the Resolution Wall via Hardware-Algorithm Co-Designcited by 1
  26. 2026Cornserve: A Distributed Serving System for Any-to-Any Multimodal Modelscited by 1
  27. 2026GF-DiT: Scheduling Parallelism for Diffusion Transformer Servingcited by 1
  28. 2026Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generationcited by 1
  29. 2025Training Video Foundation Models with NVIDIA NeMocited by 0
  30. 2025Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Executioncited by 0
  31. 2026DISK: Dynamic Inference SKipping for World Modelscited by 0
  32. 2026Accelerating Diffusion via Hybrid Data-Pipeline Parallelism Based on Conditional Guidance Schedulingcited by 0
  33. 2026StreamWise: Serving Multi-Modal Generation in Real-Time at Scalecited by 0
  34. 2026SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Servingcited by 0
  35. 2026LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflowscited by 0
  36. 2026ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inferencecited by 0
  37. 2026AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformerscited by 0
  38. 2026Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Modelscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
  2. 2024Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study
  3. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  4. 2025KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training
  5. 2024SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers
  6. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflection
  7. 2025DiffPro: Joint Timestep and Layer-Wise Precision Optimization for Efficient Diffusion Inference
  8. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas