mlsanityData to 14 Oct 2026

DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention

arXiv 2405.18428 ↗First posted 28 May 2024cs.CV cs.AIcited by 56

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion models with large-scale pre-training have achieved significant success in the field of visual content generation, particularly exemplified by Diffusion Transformers (DiT). However, DiT models have faced challenges with quadratic complexity efficiency, especially when handling long sequences. In this paper, we aim to incorporate the sub-quadratic modeling capability of Gated Linear Attention (GLA) into the 2D diffusion backbone. Specifically, we introduce Diffusion Gated Linear Attention Transformers (DiG), a simple, adoptable solution with minimal parameter overhead. We offer two variants, i,e, a plain and U-shape architecture, showing superior efficiency and competitive effectiveness. In addition to superior performance to DiT and other sub-quadratic-time diffusion models at 256 × 256 resolution, DiG demonstrates greater efficiency than these methods starting from a 512 resolution. Specifically, DiG-S/2 is 2.5× faster and saves 75.7% GPU memory compared to DiT-S/2 at a 1792 resolution. Additionally, DiG-XL/2 is 4.2× faster than the Mamba-based model at a 1024 resolution and 1.8× faster than DiT with FlashAttention-2 at a 2048 resolution. We will release the code soon. Code is released at https://github.com/hustvl/DiG.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 55 of its 70 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

55 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  2. 2022VIDM: Video Implicit Diffusion Models1 Dec
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  9. 2016Improved Techniques for Training GANscited by 2,970
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  13. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  14. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  15. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  16. 2020Rethinking Attention with Performerscited by 1,601
  17. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  18. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  19. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  20. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  21. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  22. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  23. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  24. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  25. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  26. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  27. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  28. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  29. 2021Generating Images with Sparse Representationscited by 266
  30. 2022A Survey on Generative Diffusion Modelcited by 240
  31. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  32. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  33. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  34. 2021Finetuning Pretrained Transformers into RNNscited by 96
  35. 2023Diffusion Models Without Attentioncited by 75
  36. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  37. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58
  38. 2024Scalable Diffusion Models with State Space Backbonecited by 45

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 26; 2026: 22.

56 papers.

Sort:

By area:

  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  2. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  3. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  4. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  5. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 110
  6. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 94
  7. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 87
  8. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  9. 2025Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformerscited by 51
  10. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 43
  11. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  12. 2024Autoregressive Models in Vision: A Surveycited by 39
38 more
  1. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 37
  2. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  3. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  4. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  5. 2024GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Qualitycited by 18
  6. 2025Rectifying Magnitude Neglect in Linear Attentioncited by 16
  7. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  8. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformercited by 15
  9. 2025Exploring Diffusion Transformer Designs via Graftingcited by 12
  10. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 10
  11. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 10
  12. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  13. 2026ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformerscited by 7
  14. 2026MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Headcited by 7
  15. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modelingcited by 6
  16. 2025DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modelingcited by 6
  17. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  18. 2025Attention in Diffusion Model: A Surveycited by 4
  19. 2025MobileI2V: Fast and High-Resolution Image-to-Video on Mobile Devicescited by 4
  20. 2025Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attentioncited by 3
  21. 2025UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attentioncited by 3
  22. 2026Attention Sinks in Diffusion Transformers: A Causal Analysiscited by 3
  23. 2026DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flowscited by 3
  24. 2025Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attentioncited by 2
  25. 2026Pixel-Perfect Visual Geometry Estimationcited by 2
  26. 2026MPDiT: Multi-Patch Global-to-Local Transformer Architecture For Efficient Flow Matching and Diffusion Modelcited by 2
  27. 2026EFlow: Fast Few-Step Video Generator Training from Scratch via Efficient Solution Flowcited by 2
  28. 2026SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generationcited by 2
  29. 2025Lightweight Physics-Aware Zero-Shot Ultrasound Plane-Wave Denoisingcited by 1
  30. 2026FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformerscited by 1
  31. 2025ENA: Efficient N-dimensional Attentioncited by 0
  32. 2025LowDiff: Efficient Diffusion Sampling with Low-Resolution Conditioncited by 0
  33. 2026Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generationcited by 0
  34. 2026LINA: Linear Autoregressive Image Generative Models with Continuous Tokenscited by 0
  35. 2026SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformercited by 0
  36. 2026Reviving ConvNeXt for Efficient Convolutional Diffusion Modelscited by 0
  37. 2026Linear-Time Global Visual Modeling without Explicit Attentioncited by 0
  38. 2026ElasticDiT: Efficient Diffusion Transformers via Elastic Architecture and Sparse Attention for High-Resolution Image Generation on Mobile Devicescited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers
  2. 2025VSA: Faster Video Diffusion with Trainable Sparse Attention
  3. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
  4. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Models
  5. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
  6. 2026MegaSlide-DiT: Memory-Centric Adaptation and Deformable Local Attention for Efficient Video Diffusion
  7. 2023Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models
  8. 2024CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up