mlsanityData to 14 Oct 2026

Diffusion Models Without Attention

arXiv 2311.18257 ↗First posted 30 Nov 2023cs.CV cs.LGcited by 75

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In recent advancements in high-fidelity image generation, Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a key player. However, their application at high resolutions presents significant computational challenges. Current methods, such as patchifying, expedite processes in UNet and Transformer architectures but at the expense of representational capacity. Addressing this, we introduce the Diffusion State Space Model (DiffuSSM), an architecture that supplants attention mechanisms with a more scalable state space model backbone. This approach effectively handles higher resolutions without resorting to global compression, thus preserving detailed image representation throughout the diffusion process. Our focus on FLOP-efficient architectures in diffusion training marks a significant step forward. Comprehensive evaluations on both ImageNet and LSUN datasets at two resolutions demonstrate that DiffuSSMs are on par or even outperform existing diffusion models with attention modules in FID and Inception Score metrics while significantly reducing total FLOP usage.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 66 of its 75 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

66 of 75 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Improved Techniques for Training Consistency Models22 Oct
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  3. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  4. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  5. 2023Consistency Models2 Mar
  6. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  7. 2022Scalable Diffusion Models with Transformers19 Dec
  8. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  9. 2022Diffusion Models in Vision: A Survey10 Sep
  10. 2022Classifier-Free Diffusion Guidance26 Jul
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  12. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
7 more
  1. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation10 Jun
  2. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  3. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  4. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  6. 2020Denoising Diffusion Implicit Models6 Oct
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  12. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
35 more
  1. 2016Improved Techniques for Training GANscited by 2,970
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  5. 2018Group Normalizationcited by 1,524
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  9. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  10. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  12. 2021Multi-Stage Progressive Image Restorationcited by 816
  13. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  14. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  15. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  16. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  17. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  18. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  19. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  20. 2019Deep Learning for Image Super-resolution: A Surveycited by 369
  21. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  22. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  23. 2021Generating Images with Sparse Representationscited by 266
  24. 2022Long Range Language Modeling via Gated State Spacescited by 265
  25. 2020Sparse Sinkhorn Attentioncited by 251
  26. 2021Projected GANs Converge Fastercited by 208
  27. 2022Transformer Quality in Linear Timecited by 191
  28. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  29. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  30. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  31. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  32. 2021Luna: Linear Unified Nested Attentioncited by 96
  33. 2018SpectralNet: Spectral Clustering using Deep Neural Networkscited by 94
  34. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  35. 2022Pretraining Without Attentioncited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 46; 2025: 18; 2026: 8.

75 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  4. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  5. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  6. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  7. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 112
  8. 2025DDT: Decoupled Diffusion Transformercited by 109
  9. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 102
  10. 2024The Hidden Attention of Mamba Modelscited by 91
  11. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 87
  12. 2025Align Your Flow: Scaling Continuous-Time Flow Map Distillationcited by 87
38 more
  1. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  2. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  3. 2025Long-Context State-Space Video World Modelscited by 64
  4. 2024Dynamic Diffusion Transformercited by 62
  5. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  6. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  7. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  8. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  9. 2024Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Stepcited by 47
  10. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  11. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 43
  12. 2024LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexitycited by 43
  13. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  14. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36
  15. 2024Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalizationcited by 36
  16. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 32
  17. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generationcited by 32
  18. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  19. 2024Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generationcited by 21
  20. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  21. 2024FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolutioncited by 16
  22. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  23. 2024DiC: Rethinking Conv3x3 Designs in Diffusion Modelscited by 14
  24. 2024SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spacescited by 13
  25. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  26. 2025Effective and Efficient Masked Image Generation Modelscited by 13
  27. 2025Exploring Diffusion Transformer Designs via Graftingcited by 12
  28. 2024DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generationcited by 11
  29. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 10
  30. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 10
  31. 2024You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNetcited by 9
  32. 2024MaskMamba: A Hybrid Mamba-Transformer Model for Masked Image Generationcited by 8
  33. 2024Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Modelscited by 7
  34. 2025Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulationcited by 7
  35. 2024Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Modelcited by 6
  36. 2025DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modelingcited by 6
  37. 2026Frequency-Aware Flow Matching for High-Quality Image Generationcited by 6
  38. 2024LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and Mambacited by 4

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning
  2. 2022On Distillation of Guided Diffusion Models
  3. 2026Improved Distributional Diffusion Models
  4. 2024Plug-and-Play Diffusion Distillation
  5. 2023Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
  6. 2024I-Diff: Structural Regularization for High-Fidelity Diffusion Models
  7. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion
  8. 2023Masked Diffusion Models Are Fast Distribution Learners