mlsanityData to 7 Oct 2026

Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variations

arXiv 2405.14857 ↗First posted 23 May 2024cs.CV cs.AI cs.LGcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Generating image variations, where a model produces variations of an input image while preserving the semantic context has gained increasing attention. Current image variation techniques involve adapting a text-to-image model to reconstruct an input image conditioned on the same image. We first demonstrate that a diffusion model trained to reconstruct an input image from frozen embeddings, can reconstruct the image with minor variations. Second, inspired by how text-to-image models learn from web-scale text-image pairs, we explore a new pretraining strategy to generate image variations using a large collection of image pairs. Our diffusion model textitSemantica receives a random (encoded) image from a webpage as conditional input and denoises another noisy random image from the same webpage. We carefully examine various design choices for the image encoder, given its crucial role in extracting relevant context from the input image. Once trained, textitSemantica can adaptively generate new images from a dataset by simply using images from that dataset as input. Finally, we identify limitations in standard image consistency metrics for evaluating image variations and propose alternative metrics based on few-shot generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 54 of its 65 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

54 of 65 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  2. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models25 May
  3. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation24 May
  4. 2023In-Context Learning Unlocked for Diffusion Models1 May
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  6. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  10. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
32 more
  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  2. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  4. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  5. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  6. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  7. 2021Few-shot Image Generation via Cross-domain Correspondencecited by 206
  8. 2018Transferring GANs: generating images from limited datacited by 134
  9. 2020Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANscited by 132
  10. 2020Few-shot Image Generation with Elastic Weight Consolidationcited by 122
  11. 2021Instance-Conditioned GANcited by 119
  12. 2019MineGAN: effective knowledge transfer from GANs to target domains with few imagescited by 106
  13. 2019Image Generation From Small Datasets via Batch Statistics Adaptationcited by 102
  14. 2020GAN Memory with No Forgettingcited by 84
  15. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  16. 2020Few-Shot Adaptation of Generative Adversarial Networkscited by 64
  17. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 64
  18. 2022Few Shot Generative Model Adaption via Relaxed Spatial Structural Alignmentcited by 54
  19. 2023Visual In-Context Promptingcited by 52
  20. 2022A Closer Look at Few-shot Image Generationcited by 49
  21. 2021One-Shot Generative Domain Adaptationcited by 41
  22. 2022Few-Shot Diffusion Modelscited by 37
  23. 2022Few-shot Image Generation via Adaptation-Aware Kernel Modulationcited by 33
  24. 2023Context Diffusion: In-Context Aware Image Generationcited by 20
  25. 2022HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networkscited by 19
  26. 2023A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shotcited by 19
  27. 2022When, Why, and Which Pretrained GANs Are Useful?cited by 14
  28. 2021Efficient Conditional GAN Transfer with Knowledge Propagation across Classescited by 13
  29. 2022Few-shot Image Generation via Masked Discriminationcited by 12
  30. 2023WeditGAN: Few-Shot Image Generation via Latent Space Relocationcited by 12
  31. 2023Target-Aware Generative Augmentations for Single-Shot Adaptationcited by 5
  32. 2022D3T-GAN: Data-Dependent Domain Transfer GANs for Few-shot Image Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  2. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  3. 2025TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization
  4. 2023Unsupervised Semantic Correspondence Using Stable Diffusion
  5. 2023Diversity and Diffusion: Observations on Synthetic Image Distributions with Stable Diffusion
  6. 2023Real-World Image Variation by Aligning Diffusion Inversion Chain
  7. 2026ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models
  8. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models