# Analysis of Attention in Video Diffusion Transformers

arXiv: https://arxiv.org/abs/2504.10317 · HTML: https://mlsanity.com/paper/2504.10317/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 17 of 17)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing](https://mlsanity.com/paper/2403.03431/) (2024)
- [DiTFastAttn: Attention Compression for Diffusion Transformer Models](https://mlsanity.com/paper/2406.08552/) (2024)
- [Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models](https://mlsanity.com/paper/2211.02048/) (2022)
- [Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile](https://mlsanity.com/paper/2502.06155/) (2025)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)

## Built on by (most cited first, 15 of 15)

- [Softpick: No Attention Sink, No Massive Activations with Rectified Softmax](https://mlsanity.com/paper/2504.20966/) (2025)
- [Attention (as Discrete-Time Markov) Chains](https://mlsanity.com/paper/2507.17657/) (2025)
- [CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation](https://mlsanity.com/paper/2606.09639/) (2026)
- [DiTraj: training-free trajectory control for video diffusion transformer](https://mlsanity.com/paper/2509.21839/) (2025)
- [Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence](https://mlsanity.com/paper/2512.04619/) (2025)
- [InstanceV: Instance-Level Video Generation](https://mlsanity.com/paper/2511.23146/) (2025)
- [Attention Sinks in Diffusion Transformers: A Causal Analysis](https://mlsanity.com/paper/2605.09313/) (2026)
- [Registers Matter for Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2605.16147/) (2026)
- [NABLA: Neighborhood Adaptive Block-Level Attention](https://mlsanity.com/paper/2507.13546/) (2025)
- [Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention](https://mlsanity.com/paper/2605.08453/) (2026)
- [HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion](https://mlsanity.com/paper/2605.14513/) (2026)
- [Attention of a Kiss: Exploring Attention Maps in Video Diffusion for XAIxArts](https://mlsanity.com/paper/2509.05323/) (2025)
- [MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Models](https://mlsanity.com/paper/2510.17519/) (2025)
- [SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers](https://mlsanity.com/paper/2603.02883/) (2026)
- [Temporal-Attention Head Specialization During Video Diffusion Training](https://mlsanity.com/paper/2609.31654/) (2026)

## Abstract (arXiv)

We conduct an in-depth analysis of attention in video diffusion transformers (VDiTs) and report a number of novel findings. We identify three key properties of attention in VDiTs: Structure, Sparsity, and Sinks. Structure: We observe that attention patterns across different VDiTs exhibit similar structure across different prompts, and that we can make use of the similarity of attention patterns to unlock video editing via self-attention map transfer. Sparse: We study attention sparsity in VDiTs, finding that proposed sparsity methods do not work for all VDiTs, because some layers that are seemingly sparse cannot be sparsified. Sinks: We make the first study of attention sinks in VDiTs, comparing and contrasting them to attention sinks in language models. We propose a number of future directions that can make use of our insights to improve the efficiency-quality Pareto frontier for VDiTs.