mlsanityData to 7 Oct 2026

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation

arXiv 2608.26155 ↗First posted 4 Jul 2026cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models have advanced rapidly, yet most remain English-centric, as scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of high-quality non-English image-text supervision. Although multilingual text data is abundant, naive textual fine-tuning can disrupt vision-language alignment and induce catastrophic forgetting. We propose Vision-Free Adaptation (VFA), a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. Specifically, we fine-tune a base LLM on multilingual text data to derive a multilingual task vector, which is then merged with the vision-aligned task vector of an MLLM. Experiments on five MLLMs across six multilingual multimodal benchmarks show consistent improvements while preserving both general multimodal and text-only capabilities. Moreover, using less than 2% of the text data, VFA narrows the gap to the fully multimodal-trained model, demonstrating its data efficiency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 46 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

28 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Aya Vision: Advancing the Frontier of Multilingual Multimodality13 May
  2. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization31 Mar
  3. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages21 Oct

Other references · 25

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  8. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  9. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  10. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
13 more
  1. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  2. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  3. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  4. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  5. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  6. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  7. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledgecited by 16
  8. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  9. 2025LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategycited by 10
  10. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 10
  11. 2024The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioningcited by 9
  12. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  13. 2025Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awarenesscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization
  2. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs
  3. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  4. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  5. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Models
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  8. 2024IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities