mlsanityData to 14 Oct 2026

State of the Art on Diffusion Models for Visual Computing

arXiv 2310.07204 ↗First posted 11 Oct 2023cs.AI cs.CV cs.GRcited by 127

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion-based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state-of-the-art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 289 of its 340 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

289 of 340 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision20 Jun
  2. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping8 Jun
  3. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation7 Mar
  4. 2023Consistency Models2 Mar
  5. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance11 Oct
  6. 2022GENIE: Higher-Order Denoising Diffusion Solvers11 Oct
  7. 2022On Distillation of Guided Diffusion Models6 Oct
  8. 2022Classifier-Free Diffusion Guidance26 Jul
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  12. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
3 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Implicit Models6 Oct
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 274

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  12. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
38 more
  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  13. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  14. 2023A Survey of Large Language Modelscited by 2,576
  15. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  16. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  17. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  18. 2022Video Diffusion Modelscited by 2,305
  19. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  20. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  21. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  22. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  23. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  24. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  25. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  26. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  27. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  28. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  29. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  30. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  31. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  32. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  33. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  34. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  35. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  36. 2018Demystifying MMD GANscited by 1,293
  37. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  38. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236

50 most cited of 274 All 274 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 62; 2025: 29; 2026: 12.

127 papers.

Sort:

By area:

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  2. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  3. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  4. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  5. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 318
  6. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  7. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
  8. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  9. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  10. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  11. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  12. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
38 more
  1. 2025Video World Models with Long-term Spatial Memorycited by 136
  2. 2023SparseGS: Sparse View Synthesis using 3D Gaussian Splattingcited by 134
  3. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  4. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  5. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 113
  6. 2024DiffusionPDE: Generative PDE-Solving Under Partial Observationcited by 108
  7. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 89
  8. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 83
  9. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
  10. 2024Diffusion Models in Low-Level Vision: A Surveycited by 74
  11. 2023Intrinsic Image Diffusion for Indoor Single-view Material Estimationcited by 73
  12. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  13. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  14. 2023PolyDiff: Generating 3D Polygonal Meshes with Diffusion Modelscited by 66
  15. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 63
  16. 2024Advances in 3D Generation: A Surveycited by 63
  17. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  18. 2023NeRFiller: Completing Scenes via Generative 3D Inpaintingcited by 62
  19. 2023Gaussian Shell Maps for Efficient 3D Human Generationcited by 61
  20. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 58
  21. 2024LightIt: Illumination Modeling and Control for Diffusion Modelscited by 57
  22. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  23. 2024LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generationcited by 53
  24. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 47
  25. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 46
  26. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  27. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  28. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  29. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  30. 2024Interpreting the Weight Space of Customized Diffusion Modelscited by 36
  31. 2024Image-GS: Content-Adaptive Image Representation via 2D Gaussianscited by 35
  32. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Modelscited by 30
  33. 2024Blue noise for diffusion modelscited by 29
  34. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  35. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  36. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  37. 2024HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generationcited by 21
  38. 2025Sharp Monocular View Synthesis in Less Than a Secondcited by 20

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Video Diffusion Models
  2. 2024Diffusion Models in 3D Vision: A Survey
  3. 2024Generative AI in Vision: A Survey on Models, Metrics and Applications
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications
  5. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  7. 2022Efficient Diffusion Models for Vision: A Survey
  8. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models