mlsanityData to 14 Oct 2026

X-Fusion: Introducing New Modality to Frozen Large Language Models

arXiv 2504.20996 ↗First posted 29 Apr 2025cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation. Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks. We find that incorporating understanding-focused data improves generation quality, reducing image data noise enhances overall performance, and feature alignment accelerates convergence for smaller models but has minimal impact on larger ones. Our findings provide valuable insights into building efficient unified multimodal models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 52 of its 75 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

52 of 75 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
38 more
  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  14. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  15. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  16. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  17. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  18. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  19. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  20. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  21. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  22. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  23. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  24. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  25. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  26. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  27. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  28. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  29. 2023Generating Images with Multimodal Language Modelscited by 260
  30. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  31. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  32. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  33. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  34. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  35. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 116
  36. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  37. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  38. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  2. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 21
  3. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 15
  4. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  5. 2026From Scale to Speed: Adaptive Test-Time Scaling for Image Editingcited by 8
  6. 2025Dreamland: Controllable World Creation with Simulator and Generative Modelscited by 5
  7. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
  8. 2025PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMscited by 3
  9. 2025What Happens Next? Next Scene Prediction with a Unified Video Modelcited by 2
  10. 2025VAT: Vision Action Transformer by Unlocking Full Representation of ViTcited by 1
  11. 2025Relational Visual Similaritycited by 1
  12. 2025Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaborationcited by 1
1 more
  1. 2026Octrees as an Explicit 3D Languagecited by 0