mlsanityData to 7 Oct 2026

M-VADER: A Model for Diffusion with Multimodal Context

arXiv 2212.02936 ↗First posted 6 Dec 2022cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

also in 1 more
Abstract (arXiv)

We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the generation of images specified using combinations of image and text, and combinations of multiple images. Previously, a number of successful DM image generation algorithms have been introduced that make it possible to specify the output image using a text prompt. Inspired by the success of those models, and led by the notion that language was already developed to describe the elements of visual contexts that humans find most important, we introduce an embedding model closely related to a vision-language model. Specifically, we introduce the embedding model S-MAGMA: a 13 billion parameter multimodal decoder combining components from an autoregressive vision-language model MAGMA and biases finetuned for semantic search.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 42 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

35 of 42 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
23 more
  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2016Density estimation using Real NVPcited by 2,029
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  12. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  13. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  15. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  16. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  17. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  18. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  19. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  20. 2019Deep Learning for Deepfakes Creation and Detection: A Surveycited by 102
  21. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  22. 2021Ethics and Creativity in Computer Visioncited by 15
  23. 2020A Note on Data Biases in Generative Modelscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3.

6 papers.

  1. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  2. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  3. 2023Contrastive Conditional Latent Diffusion for Audio-visual Segmentationcited by 30
  4. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusioncited by 15
  5. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Modelscited by 9
  6. 2024Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunitiescited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  2. 2024Dual Diffusion for Unified Image Generation and Understanding
  3. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
  4. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model
  5. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception
  6. 2023Generating Images with Multimodal Language Models
  7. 2026UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation
  8. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models