mlsanityData to 7 Oct 2026

Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual

arXiv 2510.26271 ↗First posted 30 Oct 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferring knowledge from larger to smaller VLMs, applying KD in multilingualism is an underexplored area. This paper presents a controlled empirical study of KD behavior across five distillation approaches, isolating their effects on cross-lingual representation consistency and downstream performance stability under model compression. We study five distillation formulations across CLIP and SigLIP2, and evaluate them on in-domain retrieval and out-of-domain visual QA. We find that some configurations preserve or even improve multilingual retrieval robustness despite halving model size, but others fail to maintain cross-task stability, exposing design-sensitive trade-offs that aggregate accuracy alone does not reveal.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 42 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

29 of 42 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
17 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  5. 2019Relational Knowledge Distillationcited by 873
  6. 2019Contrastive Representation Distillationcited by 734
  7. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  8. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  11. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  12. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  13. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  14. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  15. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  16. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  17. 2022Multilingual Representation Distillation with Contrastive Learningcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
  2. 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
  3. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
  4. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  5. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
  6. 2026From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
  7. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  8. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Models