mlsanityData to 7 Oct 2026

High-Resolution Image Synthesis with Latent Diffusion Models

arXiv 2112.10752 ↗First posted 20 Dec 2021cs.CVcited by 18,694

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .

Key facts. Cited by 18,694 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 80 of its 111 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

80 of 111 references.

Closest ancestors · 9

Same direction, newest first

  1. 2021Variational Diffusion Models1 Jul
  2. 2021Score-based Generative Modeling in Latent Space10 Jun
  3. 2021On Fast Sampling of Diffusion Probabilistic Models31 May
  4. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  5. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  6. 2021Noise Estimation for Generative Diffusion Models6 Apr
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  8. 2020Denoising Diffusion Implicit Models6 Oct
  9. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2014Conditional Generative Adversarial Netscited by 2,152
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2016Density estimation using Real NVPcited by 2,029
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  14. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  15. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  16. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  17. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  18. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  19. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  20. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  21. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  22. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  23. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  24. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  25. 2016Pixel Recurrent Neural Networkscited by 814
  26. 2016Generative Adversarial Text to Image Synthesiscited by 735
  27. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  28. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  29. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  30. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  31. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  32. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  33. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  34. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  35. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  36. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 524
  37. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  38. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 449; 2023: 3420; 2024: 5413; 2025: 5644; 2026: 3766.

18,694 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  7. 2023Consistency Modelscited by 1,778
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  13. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  14. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  15. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  16. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  17. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  18. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  19. 2024Depth Anything V2cited by 1,385
  20. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  21. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  22. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  23. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  24. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  25. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  26. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  27. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  28. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  29. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  30. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  31. 2022Diffusion Models in Vision: A Surveycited by 1,008
  32. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  33. 2022Token Merging: Your ViT But Fastercited by 995
  34. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  35. 2025Qwen-Image Technical Reportcited by 973
  36. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  37. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  38. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958

50 most cited of 18,694 All 18,694 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space
  2. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
  3. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
  4. 2022Understanding Diffusion Models: A Unified Perspective
  5. 2024Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models