mlsanityData to 13 Oct 2026

X-VILA: Cross-Modality Alignment for Large Language Model

arXiv 2405.19335 ↗First posted 29 May 2024cs.CV cs.CL cs.LGcited by 46

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities. By aligning modality-specific encoders with LLM inputs and diffusion decoders with LLM outputs, X-VILA achieves cross-modality understanding, reasoning, and generation. To facilitate this cross-modality alignment, we curate an effective interleaved any-to-any modality instruction-following dataset. Furthermore, we identify a significant problem with the current cross-modality alignment method, which results in visual information loss. To address the issue, we propose a visual alignment mechanism with a visual embedding highway module. We then introduce a resource-efficient recipe for training X-VILA, that exhibits proficiency in any-to-any modality conversation, surpassing previous approaches by large margins. X-VILA also showcases emergent properties across modalities even in the absence of similar training data. The project will be made open-source.

Key facts. Cited by 46 papers in the map (data through 13 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 67 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

56 of 67 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  7. 2023GPT-4 Technical Reportcited by 18,278
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  12. 2023Segment Anythingcited by 8,195
38 more
  1. 2023Visual Instruction Tuningcited by 7,827
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  14. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
  15. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  16. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,465
  17. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,342
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  19. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,168
  20. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,052
  21. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  22. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  23. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  24. 2017Dense-Captioning Events in Videoscited by 790
  25. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  26. 2024Yi: Open Foundation Models by 01.AIcited by 715
  27. 2023VILA: On Pre-training for Visual Language Modelscited by 690
  28. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  29. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 613
  30. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 596
  31. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 571
  32. 2021Image Segmentation Using Text and Image Promptscited by 511
  33. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  34. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  35. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  36. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  37. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  38. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 17; 2025: 23; 2026: 6.

46 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 740
  2. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 306
  3. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 244
  4. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 205
  5. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  6. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 146
  7. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 82
  8. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  9. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 55
  10. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  11. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  12. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
34 more
  1. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
  2. 2024Autoregressive Models in Vision: A Surveycited by 39
  3. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  4. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  5. 20253D Aware Region Prompted Vision Language Modelcited by 32
  6. 2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
  7. 2024VILA2: VILA Augmented VILAcited by 25
  8. 2024PUMA: Empowering Unified MLLM with Multi-granular Visual Generationcited by 20
  9. 2024Law of Vision Representation in MLLMscited by 17
  10. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 17
  11. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  12. 2025OG-VLA: Orthographic Image Generation for 3D-Aware Vision-Language Action Modelcited by 13
  13. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Modelscited by 12
  14. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  15. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 10
  16. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 9
  17. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  18. 2026Evolution of Video Generative Foundationscited by 7
  19. 2025Token Sequence Compression for Efficient Multimodal Computingcited by 6
  20. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generationcited by 5
  21. 2025DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cyclescited by 5
  22. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  23. 2025OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignmentcited by 5
  24. 2024Olympus: A Universal Task Router for Computer Vision Taskscited by 4
  25. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  26. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  27. 2026STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learningcited by 3
  28. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 3
  29. 2025Resource Consumption Red-Teaming for Large Vision-Language Modelscited by 2
  30. 2025Position: Require Frontier AI Labs To Release Small "Analog" Modelscited by 2
  31. 2026How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?cited by 2
  32. 2024Tiny-Align: Bridging Automatic Speech Recognition and Large Language Model on the Edgecited by 1
  33. 2025A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agentscited by 0
  34. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  2. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
  3. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMs
  4. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
  6. 2024ModaVerse: Efficiently Transforming Modalities with LLMs
  7. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models
  8. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages