# Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer

arXiv: https://arxiv.org/abs/2509.24899 · HTML: https://mlsanity.com/paper/2509.24899/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 45 of its 51 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Pyramidal Flow Matching for Efficient Video Generative Modeling](https://mlsanity.com/paper/2410.05954/) (2024)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [Fast Video Generation with Sliding Tile Attention](https://mlsanity.com/paper/2502.04507/) (2025)
- [The Mamba in the Llama: Distilling and Accelerating Hybrid Models](https://mlsanity.com/paper/2408.15237/) (2024)
- [The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry](https://mlsanity.com/paper/2402.04347/) (2024)
- [Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers](https://mlsanity.com/paper/2401.11605/) (2024)
- [VSA: Faster Video Diffusion with Trainable Sparse Attention](https://mlsanity.com/paper/2505.13389/) (2025)
- [DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention](https://mlsanity.com/paper/2405.18428/) (2024)
- [Linearizing Large Language Models](https://mlsanity.com/paper/2405.06640/) (2024)
- [Bridging the Divide: Reconsidering Softmax and Linear Attention](https://mlsanity.com/paper/2412.06590/) (2024)
- [LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity](https://mlsanity.com/paper/2412.09856/) (2024)
- [Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models](https://mlsanity.com/paper/2404.04478/) (2024)

## Built on by (most cited first, 15 of 15)

- [MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models](https://mlsanity.com/paper/2510.19022/) (2025)
- [Neodragon: Mobile Video Generation using Diffusion Transformer](https://mlsanity.com/paper/2511.06055/) (2025)
- [ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers](https://mlsanity.com/paper/2601.04342/) (2026)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference](https://mlsanity.com/paper/2601.04792/) (2026)
- [Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion](https://mlsanity.com/paper/2605.16579/) (2026)
- [VMonarch: Efficient Video Diffusion Transformers with Structured Attention](https://mlsanity.com/paper/2601.22275/) (2026)
- [MobileWan: Closing the Quality Gap for Mobile Video Diffusion](https://mlsanity.com/paper/2607.06173/) (2026)
- [SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation](https://mlsanity.com/paper/2607.21553/) (2026)
- [HLA: Hadamard Linear Attention](https://mlsanity.com/paper/2602.12128/) (2026)
- [FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation](https://mlsanity.com/paper/2603.09721/) (2026)
- [SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation](https://mlsanity.com/paper/2608.16585/) (2026)
- [Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion](https://mlsanity.com/paper/2609.31050/) (2026)
- [Video Generation Models: A Survey of Post-Training and Alignment](https://mlsanity.com/paper/2610.00812/) (2026)
- [LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization](https://mlsanity.com/paper/2610.17914/) (2026)

## Abstract (arXiv)

Transformer-based video diffusion models (VDMs) deliver state-of-the-art video generation quality but are constrained by the quadratic cost of self-attention, making long sequences and high resolutions computationally expensive. While linear attention offers sub-quadratic complexity, previous approaches have failed to match the expressiveness of softmax attention unless retrained at significant computational cost. We introduce Attention Surgery, an efficient framework that enables linear or hybrid attention in pretrained VDMs, eliminating the need for training from scratch. Inspired by recent advances in language models, our method combines a novel hybrid attention mechanism-mixing softmax and linear tokens-with a lightweight distillation and fine-tuning pipeline requiring only a few GPU-days. Additionally, we incorporate a cost-aware block-rate strategy to balance expressiveness and efficiency across layers. Applied to Wan2.1 1.3B, a state-of-the-art efficient transformer VDM and evaluated on VBench, VBench2.0 and a human preference study, Attention Surgery achieves competitive results. Furthermore, measurements of on-mobile latency, memory usage, and FLOPs demonstrate notable improvements in scaling behavior for longer videos. Project page is available at: https://qualcomm-ai-research.github.io/attention-surgery.