# Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study

arXiv: https://arxiv.org/abs/2411.13588 · HTML: https://mlsanity.com/paper/2411.13588/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 11 of 11)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers](https://mlsanity.com/paper/2406.01125/) (2024)
- [Real-Time Video Generation with Pyramid Attention Broadcast](https://mlsanity.com/paper/2408.12588/) (2024)
- [Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching](https://mlsanity.com/paper/2406.01733/) (2024)
- [DiTFastAttn: Attention Compression for Diffusion Transformer Models](https://mlsanity.com/paper/2406.08552/) (2024)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [Faster Diffusion via Temporal Attention Decomposition](https://mlsanity.com/paper/2404.02747/) (2024)
- [FRDiff : Feature Reuse for Universal Training-free Acceleration of Diffusion Models](https://mlsanity.com/paper/2312.03517/) (2023)

## Built on by (most cited first, 9 of 9)

- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training](https://mlsanity.com/paper/2502.07590/) (2025)
- [Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape](https://mlsanity.com/paper/2505.22918/) (2025)
- [Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas](https://mlsanity.com/paper/2506.03275/) (2025)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [M*: A Modular, Extensible, Serving System for Multimodal Models](https://mlsanity.com/paper/2606.12688/) (2026)
- [Accelerating Parallel Diffusion Model Serving with Residual Compression](https://mlsanity.com/paper/2507.17511/) (2025)
- [Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolution](https://mlsanity.com/paper/2509.23980/) (2025)
- [CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformers](https://mlsanity.com/paper/2512.24195/) (2025)

## Abstract (arXiv)

The increased model capacity of Diffusion Transformers (DiTs) and the demand for generating higher resolutions of images and videos have led to a significant rise in inference latency, impacting real-time performance adversely. While prior research has highlighted the presence of high similarity in activation values between adjacent diffusion steps (referred to as redundancy) and proposed various caching mechanisms to mitigate computational overhead, the exploration of redundancy in existing literature remains limited, with findings often not generalizable across different DiT models. This study aims to address this gap by conducting a comprehensive investigation into redundancy across a broad spectrum of mainstream DiT models. Our experimental analysis reveals substantial variations in the distribution of redundancy across diffusion steps among different DiT models. Interestingly, within a single model, the redundancy distribution remains stable regardless of variations in input prompts, step counts, or scheduling strategies. Given the lack of a consistent pattern across diverse models, caching strategies designed for a specific group of models may not easily transfer to others. To overcome this challenge, we introduce a tool for analyzing the redundancy of individual models, enabling subsequent research to develop tailored caching strategies for specific model architectures. The project is publicly available at https://github.com/xdit-project/DiTCacheAnalysis.