mlsanityData to 7 Oct 2026

Generating Images with Multimodal Language Models

arXiv 2305.17216 ↗First posted 26 May 2023cs.CL cs.CV cs.LGcited by 259

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-text inputs, and produce retrieved images, generated images, and generated text -- outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.

Key facts. Cited by 259 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 73 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

61 of 73 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  14. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  15. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  16. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  17. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  18. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  19. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  21. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  22. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  23. 2020Efficient Transformers: A Surveycited by 784
  24. 2016Generative Adversarial Text to Image Synthesiscited by 735
  25. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  26. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  27. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  28. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  29. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  30. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  31. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  32. 2016Visual Dialogcited by 377
  33. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  34. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  35. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  36. 2016Visual Storytellingcited by 176
  37. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  38. 2023A Recipe for Watermarking Diffusion Modelscited by 136

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 50; 2024: 107; 2025: 68; 2026: 34.

259 papers.

Sort:

By area:

  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  5. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  6. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  9. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  10. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  11. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
38 more
  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  2. 2025Transfer between Modalities with MetaQueriescited by 231
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  4. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  6. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  7. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  8. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  9. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  10. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  11. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  12. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  13. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  14. 2023Getting aligned on representational alignmentcited by 142
  15. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  16. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  17. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  18. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  19. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  20. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  21. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  22. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  23. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  24. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  25. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  26. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  27. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  28. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  29. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  30. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  31. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  32. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  33. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  34. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  35. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  36. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  37. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  38. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49

50 most cited of 259 All 259 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
  2. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?
  3. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  4. 2024The Revolution of Multimodal Large Language Models: A Survey
  5. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
  6. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  7. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  8. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory