# CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

arXiv: https://arxiv.org/abs/2412.16112 · HTML: https://mlsanity.com/paper/2412.16112/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 64 of its 73 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)

## Built on by (most cited first, 28 of 28)

- [Fast Video Generation with Sliding Tile Attention](https://mlsanity.com/paper/2502.04507/) (2025)
- [Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation](https://mlsanity.com/paper/2506.19852/) (2025)
- [Long-Context State-Space Video World Models](https://mlsanity.com/paper/2505.20171/) (2025)
- [Training-Free Efficient Video Generation via Dynamic Token Carving](https://mlsanity.com/paper/2505.16864/) (2025)
- [Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers](https://mlsanity.com/paper/2506.03065/) (2025)
- [DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers](https://mlsanity.com/paper/2503.22796/) (2025)
- [Bidirectional Sparse Attention for Faster Video Diffusion Training](https://mlsanity.com/paper/2509.01085/) (2025)
- [Ultra-Resolution Adaptation with Ease](https://mlsanity.com/paper/2503.16322/) (2025)
- [Turbo2K: Towards Ultra-Efficient and High-Quality 2K Video Synthesis](https://mlsanity.com/paper/2504.14470/) (2025)
- [Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers](https://mlsanity.com/paper/2505.14687/) (2025)
- [Exploring Diffusion Transformer Designs via Grafting](https://mlsanity.com/paper/2506.05340/) (2025)
- [LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation](https://mlsanity.com/paper/2501.12976/) (2025)
- [RegionE: Adaptive Region-Aware Generation for Efficient Image Editing](https://mlsanity.com/paper/2510.25590/) (2025)
- [UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios](https://mlsanity.com/paper/2511.18050/) (2025)
- [NAMI: Efficient Image Generation via Bridged Progressive Rectified Flow Transformers](https://mlsanity.com/paper/2503.09242/) (2025)
- [UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention](https://mlsanity.com/paper/2510.16325/) (2025)
- [Linearizing Vision Transformer with Test-Time Training](https://mlsanity.com/paper/2605.02772/) (2026)
- [CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers](https://mlsanity.com/paper/2605.14191/) (2026)
- [HilbertA: Hilbert Attention for Image Generation with Diffusion Models](https://mlsanity.com/paper/2509.26538/) (2025)
- [SkipSR: Faster Super Resolution with Token Skipping](https://mlsanity.com/paper/2510.08799/) (2025)
- [EDITBRIDGE: Towards Faithful and Efficient Ultra-High-Resolution Image Editing](https://mlsanity.com/paper/2608.18063/) (2026)
- [ENA: Efficient N-dimensional Attention](https://mlsanity.com/paper/2508.11921/) (2025)
- [MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching](https://mlsanity.com/paper/2510.14260/) (2025)
- [HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Models](https://mlsanity.com/paper/2608.07003/) (2026)
- [Beyond Attention Masks: Instruction Anchoring for Efficient In-Context Diffusion Generation](https://mlsanity.com/paper/2608.21229/) (2026)
- [Copy the Same, Distill the Difference: Initializing Linear Vision Transformers](https://mlsanity.com/paper/2609.35745/) (2026)
- [Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation](https://mlsanity.com/paper/2610.08772/) (2026)
- [LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization](https://mlsanity.com/paper/2610.17914/) (2026)

## Abstract (arXiv)

Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization cross various models and plugins, and improved support for multi-GPU parallel inference. Models and codes are available here: https://github.com/Huage001/CLEAR.