mlsanityData to 7 Oct 2026

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

arXiv 2303.06555 ↗First posted 12 Mar 2023cs.LG cs.CVcited by 185

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Our key insight is -- learning diffusion models for marginal, conditional, and joint distributions can be unified as predicting the noise in the perturbed data, where the perturbation levels (i.e. timesteps) can be different for different modalities. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model -- perturbs data in all modalities instead of a single modality, inputs individual timesteps in different modalities, and predicts the noise of all modalities instead of a single modality. UniDiffuser is parameterized by a transformer for diffusion models to handle input types of different modalities. Implemented on large-scale paired image-text data, UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead. In particular, UniDiffuser is able to produce perceptually realistic samples in all tasks and its quantitative results (e.g., the FID and CLIP score) are not only superior to existing general-purpose models but also comparable to the bespoken models (e.g., Stable Diffusion and DALL-E 2) in representative tasks (e.g., text-to-image generation).

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 50 of its 58 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

50 of 58 references.

Closest ancestors · 19

Same direction, newest first

  1. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  2. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning8 Aug
  3. 2022Classifier-Free Diffusion Guidance26 Jul
  4. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations14 Jul
  5. 2022Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching16 Jun
  6. 2022Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models15 Jun
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  9. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
  12. 2021Variational Diffusion Models1 Jul
7 more
  1. 2021Score-based Generative Modeling in Latent Space10 Jun
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  4. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  6. 2020Denoising Diffusion Implicit Models6 Oct
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022Video Diffusion Modelscited by 2,280
19 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  13. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  14. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  15. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  16. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  17. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  18. 2022Equivariant Energy-Guided SDE for Inverse Molecular Designcited by 72
  19. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 45; 2024: 67; 2025: 41; 2026: 32.

185 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  2. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  3. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  5. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  6. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  7. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 159
  8. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  9. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  10. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  11. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  12. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
38 more
  1. 2023DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generationcited by 110
  2. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  3. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  4. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  5. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
  6. 2023AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondencecited by 61
  7. 2023The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editingcited by 61
  8. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  9. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 59
  10. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  11. 2026Motubrain: An Advanced World Action Model for Robot Controlcited by 54
  12. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  13. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  14. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 48
  15. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  16. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  17. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  18. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  19. 2024OctFusion: Octree-based Diffusion Models for 3D Shape Generationcited by 40
  20. 2024Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancycited by 38
  21. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 37
  22. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  23. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  24. 2023BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledgecited by 30
  25. 2023UGG: Unified Generative Graspingcited by 30
  26. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  27. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Modelscited by 29
  28. 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesiscited by 29
  29. 2024Unimotion: Unifying 3D Human Motion Synthesis and Understandingcited by 29
  30. 2024Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equationscited by 28
  31. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 27
  32. 2023Toward Understanding Generative Data Augmentationcited by 25
  33. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  34. 2023LLMGA: Multimodal Large Language Model based Generation Assistantcited by 25
  35. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformercited by 24
  36. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
  37. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 22
  38. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers
  2. 2022Exploring Transformer Backbones for Image Diffusion Models
  3. 2023Neural Diffusion Models
  4. 2024Diffusion Models For Multi-Modal Generative Modeling
  5. 2026UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception
  6. 2022Scalable Diffusion Models with Transformers
  7. 2024UniFL: Improve Latent Diffusion Model via Unified Feedback Learning
  8. 2026Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models