mlsanityData to 14 Oct 2026

Text-driven Visual Synthesis with Latent Diffusion Prior

arXiv 2302.08510 ↗First posted 16 Feb 2023cs.CVcited by 6

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

There has been tremendous progress in large-scale text-to-image synthesis driven by diffusion models enabling versatile downstream applications such as 3D object synthesis from texts, image editing, and customized generation. We present a generic approach using latent diffusion models as powerful image priors for various visual synthesis tasks. Existing methods that utilize such priors fail to use these models' full capabilities. To improve this, our core ideas are 1) a feature matching loss between features from different layers of the decoder to provide detailed guidance and 2) a KL divergence loss to regularize the predicted latent features and stabilize the training. We demonstrate the efficacy of our approach on three different applications, text-to-3D, StyleGAN adaptation, and layered image editing. Extensive results show our method compares favorably against baselines.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

52 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Diffusion Guided Domain Adaptation of Image Generators8 Dec
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  3. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
32 more
  1. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  2. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  3. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  4. 2022Denoising Diffusion Restoration Modelscited by 963
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  6. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  7. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  8. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  9. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  10. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  11. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  12. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  13. 2022Diffusion models as plug-and-play priorscited by 231
  14. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  15. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  16. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  17. 2021Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesiscited by 140
  18. 2018Transferring GANs: generating images from limited datacited by 134
  19. 2020Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANscited by 132
  20. 2020Few-shot Image Generation with Elastic Weight Consolidationcited by 122
  21. 2020Structure-Preserving Super Resolution with Gradient Guidancecited by 116
  22. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 107
  23. 2019Image Generation From Small Datasets via Batch Statistics Adaptationcited by 102
  24. 2020Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domainscited by 96
  25. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  26. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  27. 2020Few-Shot Adaptation of Generative Adversarial Networkscited by 64
  28. 2021Data-Efficient Instance Generation from Instance Discriminationcited by 58
  29. 2021ClipMatrix: Text-controlled Creation of 3D Textured Meshescited by 40
  30. 2022Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encodingcited by 39
  31. 2021Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Modelcited by 34
  32. 2022Temporally Consistent Semantic Video Editingcited by 28

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 1.

6 papers.

  1. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  2. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 36
  3. 2023Collage Diffusioncited by 33
  4. 2023DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Viewscited by 31
  5. 2023Diffusion-based Image Translation with Label Guidance for Domain Adaptive Semantic Segmentationcited by 27
  6. 2024GLoD: Composing Global Contexts and Local Details in Image Generationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  2. 2022Blended Latent Diffusion
  3. 2023PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion
  4. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion
  5. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis
  6. 2026TextLDM: Language Modeling with Continuous Latent Diffusion
  7. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  8. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis