mlsanityData to 7 Oct 2026

Hierarchical Text-Conditional Image Generation with CLIP Latents

arXiv 2204.06125 ↗First posted 13 Apr 2022cs.CVcited by 6,539

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.

Key facts. Cited by 6,539 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 50 of its 66 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

50 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2022Generative Adversarial Networkscited by 5,603
37 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  13. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  14. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  15. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  16. 2021Score-based Generative Modeling in Latent Spacecited by 634
  17. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 524
  18. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  19. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  20. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  21. 2021GAN Inversion: A Surveycited by 366
  22. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  23. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  24. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  25. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  26. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  27. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  28. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  29. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  30. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  31. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  32. 2020Learning Visual Representations with Caption Annotationscited by 123
  33. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  34. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 64
  35. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  36. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  37. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 538; 2023: 2020; 2024: 2042; 2025: 1367; 2026: 572.

6,539 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2023Consistency Modelscited by 1,778
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  13. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  14. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  15. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  16. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  17. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  18. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  19. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  20. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  21. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  22. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  23. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  24. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  25. 2022Human Motion Diffusion Modelcited by 1,062
  26. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  27. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  28. 2022Diffusion Models in Vision: A Surveycited by 1,008
  29. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  30. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  31. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  32. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  33. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  34. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  35. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  36. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  37. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  38. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914

50 most cited of 6,539 All 6,539 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation
  2. 2024Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings
  3. 2024ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model
  4. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
  5. 2024CLIP-VQDiffusion : Langauge Free Training of Text To Image generation using CLIP and vector quantized diffusion model
  6. 2025CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation
  7. 2026Text-to-Image Models Need Less from Text Encoders Than You Think
  8. 2023CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts