mlsanityData to 7 Oct 2026

Unified Discrete Diffusion for Simultaneous Vision-Language Generation

arXiv 2211.14842 ↗First posted 27 Nov 2022cs.CVcited by 39

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The recently developed discrete diffusion models perform extraordinarily well in the text-to-image task, showing significant promise for handling the multi-modality signals. In this work, we harness these traits and present a unified multimodal generation model that can conduct both the "modality translation" and "multi-modality generation" tasks using a single model, performing text-based, image-based, and even vision-language simultaneous generation. Specifically, we unify the discrete diffusion process for multimodal signals by proposing a unified transition matrix. Moreover, we design a mutual attention module with fused embedding layer and a unified objective function to emphasise the inter-modal linkages, which are vital for multi-modality generation. Extensive experiments indicate that our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 47 of its 56 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

47 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  3. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
32 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  7. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  12. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  14. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  15. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  16. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  17. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  18. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  19. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  20. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  21. 2019Attention on Attention for Image Captioningcited by 289
  22. 2020X-Linear Attention Networks for Image Captioningcited by 198
  23. 2019Image Captioning: Transforming Objects into Wordscited by 171
  24. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  25. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  26. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  27. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  28. 2020Latent Video Transformercited by 106
  29. 2022Improved Vector Quantized Diffusion Modelscited by 62
  30. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  31. 2021Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generationcited by 40
  32. 2021L-Verse: Bidirectional Generation Between Image and Textcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 4; 2025: 14; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  3. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  4. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  5. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  6. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  7. 2023Online Clustered Codebookcited by 55
  8. 2025Unified Multimodal Discrete Diffusioncited by 46
  9. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  10. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 43
  11. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  12. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
27 more
  1. 2023Vector Quantized Wasserstein Auto-Encodercited by 22
  2. 2025Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spacescited by 19
  3. 2023PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editorcited by 17
  4. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 14
  5. 2023Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generationcited by 12
  6. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generationcited by 12
  7. 2023CoBIT: A Contrastive Bi-directional Image-Text Generation Modelcited by 11
  8. 2023Multi-modal Latent Diffusioncited by 9
  9. 2025Pulp Motion: Framing-aware multimodal camera and human motion generationcited by 8
  10. 2023One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controlscited by 6
  11. 2025Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Modelscited by 6
  12. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
  13. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  14. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  15. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  16. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  17. 2024Covariance-Adaptive Sequential Black-box Optimization for Diffusion Targeted Generationcited by 3
  18. 2023OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML Systemcited by 2
  19. 2024Ground-A-Score: Scaling Up the Score Distillation for Multi-Attribute Editingcited by 2
  20. 2026Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestepcited by 2
  21. 2025Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generationcited by 1
  22. 2025ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Modelscited by 1
  23. 2026SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generationcited by 1
  24. 2026One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iterationcited by 1
  25. 2025MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Modelscited by 0
  26. 2026Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesiscited by 0
  27. 2026BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual Diffusion for Unified Image Generation and Understanding
  2. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
  3. 2026UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation
  4. 2025MMGen: Unified Multi-modal Image Generation and Understanding in One Go
  5. 2023Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation
  6. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
  7. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
  8. 2024Diffusion Models For Multi-Modal Generative Modeling