mlsanityData to 7 Oct 2026

Versatile Diffusion: Text, Images and Variations All in One Diffusion Model

arXiv 2211.08332 ↗First posted 15 Nov 2022cs.CVcited by 199

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent advances in diffusion models have set an impressive milestone in many generation tasks, and trending works such as DALL-E2, Imagen, and Stable Diffusion have attracted great interest. Despite the rapid landscape changes, recent new approaches focus on extensions and performance rather than capacity, thus requiring separate models for separate tasks. In this work, we expand the existing single-flow diffusion pipeline into a multi-task multimodal network, dubbed Versatile Diffusion (VD), that handles multiple flows of text-to-image, image-to-text, and variations in one unified model. The pipeline design of VD instantiates a unified multi-flow diffusion framework, consisting of sharable and swappable layer modules that enable the crossmodal generality beyond images and text. Through extensive experiments, we demonstrate that VD successfully achieves the following: a) VD outperforms the baseline approaches and handles all its base tasks with competitive quality; b) VD enables novel extensions such as disentanglement of style and semantics, dual- and multi-context blending, etc.; c) The success of our multi-flow multimodal framework over images and text may inspire further diffusion-based universal AI research. Our code and models are open-sourced at https://github.com/SHI-Labs/Versatile-Diffusion.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 87 of its 117 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

87 of 117 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  11. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  12. 2020Denoising Diffusion Implicit Modelscited by 9,655
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2017Mask R-CNNcited by 7,259
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  13. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  14. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  15. 2015VQA: Visual Question Answeringcited by 2,489
  16. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  17. 2015Variational Inference with Normalizing Flowscited by 2,161
  18. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  19. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  20. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  21. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  22. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  23. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  24. 2018Group Normalizationcited by 1,514
  25. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  26. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  27. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  28. 2019Contrastive Multiview Codingcited by 1,297
  29. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  30. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  31. 2021Variational Diffusion Modelscited by 1,145
  32. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  33. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  34. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  35. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  36. 2016Pixel Recurrent Neural Networkscited by 814
  37. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  38. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 56; 2024: 63; 2025: 58; 2026: 21.

199 papers.

Sort:

By area:

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  5. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  6. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  8. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  9. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  10. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  11. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  12. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
38 more
  1. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  2. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  3. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  4. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  5. 2024Transparent Image Layer Diffusion using Latent Transparencycited by 111
  6. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  7. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  8. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  9. 2024IMAGDressing-v1: Customizable Virtual Dressingcited by 100
  10. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  11. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  12. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  13. 2023Circumventing Concept Erasure Methods For Text-to-Image Generative Modelscited by 81
  14. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  15. 2023HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Modelscited by 67
  16. 2024MindBridge: A Cross-Subject Brain Decoding Frameworkcited by 65
  17. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
  18. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  19. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  20. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  21. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 55
  22. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 48
  23. 2023On the De-duplication of LAION-2Bcited by 47
  24. 2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
  25. 2023DREAM: Visual Decoding from Reversing Human Visual Systemcited by 44
  26. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 43
  27. 2023Perceptual Artifacts Localization for Image Synthesis Taskscited by 43
  28. 2023UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activitycited by 42
  29. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  30. 2023Generative Semantic Communication: Diffusion Models Beyond Bit Recoverycited by 41
  31. 2024One Diffusion to Generate Them Allcited by 41
  32. 2025Efficient Diffusion Models: A Surveycited by 35
  33. 2024NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulationcited by 33
  34. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  35. 2024UMBRAE: Unified Multimodal Brain Decodingcited by 31
  36. 2023UGG: Unified Generative Graspingcited by 30
  37. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30
  38. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models
  2. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  3. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
  4. 2023Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations
  5. 2024Contextualized Diffusion Models for Text-Guided Image and Video Generation
  6. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  7. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decomposition
  8. 2023De-Diffusion Makes Text a Strong Cross-Modal Interface