mlsanityData to 7 Oct 2026

X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

arXiv 2403.11399 ↗First posted 18 Mar 2024cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The impressive development of large language models (LLMs) is expanding into the realm of large multimodal models (LMMs), which incorporate multiple types of data beyond text. However, the nature of multimodal models leads to significant expenses in the creation of training data. Furthermore, constructing multilingual data for LMMs presents its own set of challenges due to language diversity and complexity. Therefore, in this study, we propose two cost-effective methods to solve this problem: (1) vocabulary expansion and pretraining of multilingual LLM for specific languages, and (2) automatic and elaborate construction of multimodal datasets using GPT4-V. Based on015 these methods, we constructed a 91K English-Korean-Chinese multilingual, multimodal training dataset. Additionally, we developed a bilingual multimodal model that exhibits excellent performance in both Korean and English, surpassing existing approaches.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 33 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

24 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning12 Oct
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023Visual Instruction Tuning with Polite Flamingo3 Jul
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
6 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2019How multilingual is Multilingual BERT?cited by 642
  3. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  4. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  5. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  6. 2024BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretrainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 1.

10 papers.

  1. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  2. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  3. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  4. 2024VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generationcited by 3
  5. 2026Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Frameworkcited by 3
  6. 2025SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winnercited by 2
  7. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  8. 2025Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learningcited by 1
  9. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspectivecited by 0
  10. 2025Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
  2. 2024Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages
  3. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
  4. 2026M-MiniGPT4: Multilingual VLLM Alignment via Translated Data
  5. 2024A Review of Multi-Modal Large Language and Vision Models
  6. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs
  7. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  8. 2025Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey