mlsanityData to 7 Oct 2026

Parrot: Multilingual Visual Instruction Tuning

arXiv 2406.02539 ↗First posted 4 Jun 2024cs.CV cs.AI cs.CLcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The rapid development of Multimodal Large Language Models (MLLMs), such as GPT-4o, marks a significant step toward artificial general intelligence. Existing methods typically align vision encoders with LLMs via supervised fine-tuning (SFT), but this often deteriorates their ability to handle multiple languages as training progresses. We empirically observe that imbalanced SFT datasets, largely English-centric, degrade performance on non-English languages due to the failure in multilingual token alignment. To address this, we propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level. PARROT conditions visual tokens on diverse language inputs and uses Mixture-of-Experts (MoE) to align multilingual tokens. By computing cross-attention between initial visual features and textual embeddings, we select the most relevant experts, converting visual tokens into language-specific representations. Additionally, we introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions, to assess multilingual capabilities. PARROT achieves state-of-the-art performance on both the multilingual benchmarks and a wide range of multimodal tasks. Code and dataset are available at: https://github.com/AIDC-AI/Parrot

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 89 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

57 of 89 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  2. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  7. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning12 Oct
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023A Survey on Multimodal Large Language Models23 Jun
3 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
30 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  16. 2024Yi: Open Foundation Models by 01.AIcited by 706
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  18. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  19. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  20. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  21. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  22. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  23. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  24. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  25. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  26. 2023Model Spider: Learning to Rank Pre-Trained Models Efficientlycited by 32
  27. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  28. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selectioncited by 20
  29. 2023Learning Language-Specific Layers for Multilingual Machine Translationcited by 13
  30. 2022Identifying Ambiguous Similarity Conditions via Semantic Matchingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 16; 2026: 9.

30 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  5. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  6. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  7. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  9. 2025Multimodal Tabular Reasoning with Privileged Structured Informationcited by 14
  10. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  11. 2025C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generationcited by 9
  12. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
18 more
  1. 2025Information Density Principle for MLLM Benchmarkscited by 8
  2. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  3. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 7
  4. 2025KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contextscited by 6
  5. 2026Test-Time Attention Purification for Backdoored Large Vision Language Modelscited by 5
  6. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  7. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3
  8. 2025Benchmarking Vision Language Models on German Factual Datacited by 2
  9. 2025VLM@school -- Evaluation of AI image understanding on German middle school knowledgecited by 2
  10. 2024Enhancing Multimodal Large Language Models Complex Reason via Similarity Computationcited by 1
  11. 2025LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQAcited by 1
  12. 2025jina-vlm: Small Multilingual Vision Language Modelcited by 1
  13. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuningcited by 1
  14. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  15. 2026Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?cited by 1
  16. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 1
  17. 2026XPERT: Expert Knowledge Transfer for Effective Training of Language Modelscited by 0
  18. 2026Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets
  2. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
  3. 2025Visual Representation Alignment for Multimodal Large Language Models
  4. 2024Beyond Visual Understanding: Introducing PARROT-360V for Vision Language Model Benchmarking
  5. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  6. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization
  7. 2023VIGC: Visual Instruction Generation and Correction
  8. 2025Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM