mlsanityData to 14 Oct 2026

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

arXiv 2305.04160 ↗First posted 7 May 2023cs.CL cs.AI cs.CVcited by 128

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable language abilities. GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities beyond previous visual language models. We attribute this to the use of more advanced LLMs compared with previous multimodal models. Unfortunately, the model architecture and training strategies of GPT-4 are unknown. To endow LLMs with multimodal capabilities, we propose X-LLM, which converts Multi-modalities (images, speech, videos) into foreign languages using X2L interfaces and inputs them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X'' denotes multi-modalities such as image, speech, and videos, and ``L'' denotes languages. X-LLM's training consists of three stages: (1) Converting Multimodal Information: The first stage trains each X2L interface to align with its respective single-modal encoder separately to convert multimodal information into languages. (2) Aligning X2L representations with the LLM: single-modal encoders are aligned with the LLM through X2L interfaces independently. (3) Integrating multiple modalities: all single-modal encoders are aligned with the LLM through X2L interfaces to integrate multimodal capabilities into the LLM. Our experiments show that X-LLM demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 84.5% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. And we also conduct quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote the era of LLM-based speech recognition.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 53 of its 62 references that are in the map. First posted on arXiv 7 May 2023.

Builds on

53 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models30 Jan

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2015Fast R-CNNcited by 2,933
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  12. 2021Transformers in Vision: A Surveycited by 1,013
  13. 2021Scaling Vision Transformerscited by 999
  14. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  15. 2017Dense-Captioning Events in Videoscited by 793
  16. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  17. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  18. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  19. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  20. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  21. 2016Visual Dialogcited by 377
  22. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  23. 2020Rethinking Pre-training and Self-trainingcited by 352
  24. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  25. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  26. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  27. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  28. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  29. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  30. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  31. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  32. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  33. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 61
  34. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 55
  35. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  36. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42
  37. 2022Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selectioncited by 39
  38. 2020CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognitioncited by 21

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 39; 2024: 48; 2025: 35; 2026: 6.

128 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  9. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  11. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
38 more
  1. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  2. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  3. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  4. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  5. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  8. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  9. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  10. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  11. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  12. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  13. 2023Embodied Task Planning with Large Language Modelscited by 84
  14. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  15. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  16. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  17. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  18. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  19. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  20. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  21. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  22. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  23. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  24. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  25. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  26. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 47
  27. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  28. 2024Foundation Models for Video Understanding: A Surveycited by 46
  29. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  30. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  31. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 35
  32. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  33. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 31
  34. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  35. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  36. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehensioncited by 27
  37. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  38. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Review of Multi-Modal Large Language and Vision Models
  2. 2024X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment
  3. 2023NExT-GPT: Any-to-Any Multimodal LLM
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
  5. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  7. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputs
  8. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques