mlsanityData to 7 Oct 2026

Why do LLaVA Vision-Language Models Reply to Images in English?

arXiv 2407.02333 ↗First posted 2 Jul 2024cs.CL cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models' internal representations of image and text inputs. Both approaches indicate that the issue stems in the language modelling component of the LLaVA model. Statistically, we find that switching the language backbone for a bilingual language model has the strongest effect on reducing this error. Mechanistically, we provide compelling evidence that visual inputs are not mapped to a similar space as text ones, and that intervening on intermediary attention layers can reduce this bias. Our findings provide important insights to researchers and engineers seeking to understand the crossover between multimodal and multilingual spaces, and contribute to the goal of developing capable and inclusive VLMs for non-English contexts.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 52 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

38 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
25 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  3. 2023Steering Language Models With Activation Engineeringcited by 824
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  6. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  7. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  8. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  9. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  10. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  11. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  12. 2022Linearly Mapping from Image to Text Spacecited by 139
  13. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  14. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  15. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  16. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  17. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  18. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 37
  19. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  20. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  21. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  22. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16
  23. 2024LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Modelcited by 14
  24. 2024X-LLaVA: Optimizing Bilingual Large Vision-Language Alignmentcited by 10
  25. 2024What Is Missing in Multilingual Visual Reasoning and How to Fix Itcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 5.

11 papers.

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  2. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  3. 2024Steering Large Language Models to Evaluate and Amplify Creativitycited by 4
  4. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  5. 2025Behind Maya: Building a Multilingual Vision Language Modelcited by 3
  6. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptationcited by 3
  7. 2026Greater accessibility can amplify discrimination in generative AIcited by 1
  8. 2024Debias your Large Multi-Modal Model at Test-Time via Non-Contrastive Visual Attribute Steeringcited by 0
  9. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspectivecited by 0
  10. 2026"Inctelegi Romanecste?'' A Recipe for Romanian Vision-Language Modelscited by 0
  11. 2026One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
  2. 2024Can We Talk Models Into Seeing the World Differently?
  3. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2026Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework
  6. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding
  7. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  8. 2025Rethinking Visual Information Processing in Multimodal LLMs