mlsanityData to 14 Oct 2026

DreamLLM: Synergistic Multimodal Comprehension and Creation

arXiv 2309.11499 ↗First posted 20 Sep 2023cs.CV cs.CL cs.LGcited by 297

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io.

Key facts. Cited by 297 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 154 of its 188 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

154 of 188 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023Generating Images with Multimodal Language Models26 May
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 146

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  12. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
38 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
  13. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  14. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  15. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  16. 2017Neural Discrete Representation Learningcited by 4,712
  17. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  18. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  19. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  20. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  21. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  22. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  23. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  24. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  25. 2017Deep reinforcement learning from human preferencescited by 3,912
  26. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  27. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  28. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  29. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  30. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  31. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  32. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  33. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  34. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  35. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  36. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  37. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  38. 2015VQA: Visual Question Answeringcited by 2,506

50 most cited of 146 All 146 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 103; 2025: 117; 2026: 55.

297 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  7. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  8. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  9. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  10. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  12. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
38 more
  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  2. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  3. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  7. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  8. 2025Transfer between Modalities with MetaQueriescited by 236
  9. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  10. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  11. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  12. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  13. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  14. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  15. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  16. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  17. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  18. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  19. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  20. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  21. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  22. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  23. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  24. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  25. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
  26. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  27. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  28. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  29. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
  30. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  31. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  32. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  33. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  34. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  35. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  36. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  37. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  38. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  2. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  5. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  7. 2024DreamLIP: Language-Image Pre-training with Long Captions
  8. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Models