mlsanityData to 14 Oct 2026

LMFusion: Adapting Pretrained Language Models for Multimodal Generation

arXiv 2412.15188 ↗First posted 19 Dec 2024cs.CL cs.AI cs.CVcited by 120

Not on the map. Why?

Cited by 25 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present LMFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. LMFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules for processing images with diffusion. During training, the data from each modality is routed to its dedicated modules: modality-specific feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features. By freezing the text-specific modules and only training the image-specific modules, LMFusion preserves the language capabilities of text-only LLMs while developing strong visual understanding and generation abilities. Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LMFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs while maintaining Llama-3's language capabilities. We also demonstrate that this framework can adapt existing vision-language models with multimodal generation ability. Overall, this framework not only leverages existing computational investments in text-only LLMs but also enables the parallel development of language and vision capabilities, presenting a promising direction for efficient multimodal model development.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Builds on 50 of its 60 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

50 of 60 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  16. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  17. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  18. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  19. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  22. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  23. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  24. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  25. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  26. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  27. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  28. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  29. 2023Generating Images with Multimodal Language Modelscited by 260
  30. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  31. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  32. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  33. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  34. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 116
  35. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  36. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  37. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesiscited by 44
  38. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 74; 2026: 46.

120 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  3. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  4. 2025Transfer between Modalities with MetaQueriescited by 236
  5. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  6. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  7. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  8. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 146
  9. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  10. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 92
  11. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  12. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
38 more
  1. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  2. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 58
  3. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
  4. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  5. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  6. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  7. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  8. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 46
  9. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  10. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 41
  11. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  12. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 32
  13. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 32
  14. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 30
  15. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  16. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  17. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 26
  18. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 24
  19. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 22
  20. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  21. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 19
  22. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 18
  23. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 18
  24. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  25. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  26. 2025UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learningcited by 16
  27. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesiscited by 16
  28. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  29. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 15
  30. 2025Query-Kontext: An Unified Multimodal Model for Image Generation and Editingcited by 15
  31. 2025Towards Enhanced Image Generation Via Multi-modal Chain of Thought in Unified Generative Modelscited by 14
  32. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editingcited by 14
  33. 2026Demystifying Video Reasoningcited by 14
  34. 2025X-Fusion: Introducing New Modality to Frozen Large Language Modelscited by 13
  35. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  36. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Modelscited by 12
  37. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 12
  38. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generators
  2. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  3. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  4. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
  5. 2023LLMGA: Multimodal Large Language Model based Generation Assistant
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  8. 2025LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation