mlsanityData to 14 Oct 2026

Language-Conditioned Visual Grounding with CLIP Multilingual

arXiv 2605.09060 ↗First posted 9 May 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multilingual vision-language models exhibit systematic performance gaps across languages, but the mechanism remains ambiguous: cross-language divergence could arise from the visual encoder, the text branch, or their interaction. We resolve this ambiguity through a dense multilingual CLIP probe in which the visual encoder is held identical across thirteen typologically diverse languages and only the XLM-RoBERTa text branch varies. We evaluate two CLIP architectures spanning a 7x visual-encoder scale gap (XLM-R base + ViT-B/32, ~87M visual parameters; XLM-R large + ViT-H/14, ~632M) on 11 concepts and 210 images, and quantify cross-language agreement via cluster-mask IoU, top-percentile IoU, and Spearman rank correlation against an English reference (n=2,310 paired observations per language). Three findings emerge. First, low-resource languages (Arabic, Basque, Luxembourgish) incur a structural penalty at both backbone scales (Wilcoxon HR>LR p<10^-300; cluster-mask IoU gap +0.114 at base, +0.143 at large), isolating the deficit to the text branch. Second, scaling the encoder 7x widens the gap for structural failure cases (Basque Delta=-0.056, Luxembourgish Delta=-0.076) while improving Arabic (Delta=+0.033), separating corpus-coverage from tokeniser-fertility failures. Third, peak similarity is preserved across languages (mean ratio 0.94 at large scale) while cluster-mask IoU drops sharply, identifying spatial misalignment, not signal collapse, as the dominant failure mode. At 3.4-3.9 Wh per 1,000 queries, dense-CLIP grounding is competitive with high-throughput inference budgets, positioning it as a practical substrate for energy-aware multilingual deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 31 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

24 of 31 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2019Green AIcited by 627
  12. 2021Extract Free Dense Labels from CLIPcited by 583
12 more
  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  2. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  3. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  4. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  5. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  6. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  7. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  8. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  9. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
  10. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 111
  11. 2024One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigationcited by 13
  12. 2025Cross-Platform Evaluation of Reasoning Capabilities in Foundation Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  2. 2024A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene
  3. 2025TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
  4. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
  5. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  6. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  7. 2025Multilingual Vision-Language Models, A Survey
  8. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation