mlsanityData to 7 Oct 2026

Dual Diffusion for Unified Image Generation and Understanding

arXiv 2501.00289 ↗First posted 31 Dec 2024cs.CV cs.AI cs.LGcited by 76

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

Diffusion models have gained tremendous success in text-to-image generation, yet still lag behind with visual understanding tasks, an area dominated by autoregressive vision-language models. We propose a large-scale and fully end-to-end diffusion model for multi-modal understanding and generation that significantly improves on existing diffusion-based multimodal models, and is the first of its kind to support the full suite of vision-language modeling capabilities. Inspired by the multimodal diffusion transformer (MM-DiT) and recent advances in discrete diffusion language modeling, we leverage a cross-modal maximum likelihood estimation framework that simultaneously trains the conditional likelihoods of both images and text jointly under a single loss function, which is back-propagated through both branches of the diffusion transformer. The resulting model is highly flexible and capable of a wide range of tasks including image generation, captioning, and visual question answering. Our model attained competitive performance compared to recent unified image understanding and generation models, demonstrating the potential of multimodal diffusion modeling as a promising alternative to autoregressive next-token prediction models.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 87 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

68 of 87 references.

References · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2024The Llama 3 Herd of Modelscited by 14,192
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  13. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  14. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  15. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  18. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  20. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  21. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  22. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  23. 2021Variational Diffusion Modelscited by 1,145
  24. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  25. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  26. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  27. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  28. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  29. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  30. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  31. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  32. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  33. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  34. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  35. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  36. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  37. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  38. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 48; 2026: 28.

76 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  2. 2025Motus: A Unified Latent Action World Modelcited by 231
  3. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  4. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  5. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  6. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  7. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  8. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  9. 2025A Survey on Diffusion Language Modelscited by 75
  10. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  11. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  12. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
38 more
  1. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  2. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  3. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  4. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  5. 2025ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RLcited by 30
  6. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  7. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  8. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  9. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  10. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Studycited by 14
  11. 2025MM-ACT: Learn from Multimodal Parallel Generation to Actcited by 12
  12. 2025UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generationcited by 11
  13. 2025BWCache: Accelerating Video Diffusion Transformers through Block-Wise Cachingcited by 11
  14. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  15. 2025From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Modelcited by 9
  16. 2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
  17. 2026UEval: A Benchmark for Unified Multimodal Generationcited by 9
  18. 2025Pulp Motion: Framing-aware multimodal camera and human motion generationcited by 8
  19. 2025EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecturecited by 8
  20. 2026LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Modelcited by 8
  21. 2025On Fairness of Unified Multimodal Large Language Model for Image Generationcited by 6
  22. 2025Discrete Guidance Matching: Exact Guidance for Discrete Flow Matchingcited by 6
  23. 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 6
  24. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  25. 2025Jodi: Unification of Visual Generation and Understanding via Joint Modelingcited by 5
  26. 2025JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generationcited by 5
  27. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  28. 2025UniF2ace: A Unified Fine-grained Face Understanding and Generation Modelcited by 4
  29. 2025A Unified Multi-Agent Framework for Universal Multimodal Understanding and Generationcited by 4
  30. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  31. 2025Conditioning Residuals for Diffusion Models via Representation Feedbackcited by 3
  32. 2025From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generationcited by 3
  33. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generationcited by 3
  34. 2025End-to-End Multi-Modal Diffusion Mambacited by 3
  35. 2025MOS: Mitigating Optical-SAR Modality Gap for Cross-Modal Ship Re-Identificationcited by 3
  36. 2026PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Taskscited by 3
  37. 2026UniGenDet: A Unified Generative-Discriminative Framework for Co-Evolutionary Image Generation and Generated Image Detectioncited by 3
  38. 2026G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Modelscited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generation
  2. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
  3. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  4. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
  5. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception
  6. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
  7. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  8. 2023De-Diffusion Makes Text a Strong Cross-Modal Interface