mlsanityData to 14 Oct 2026

Seeing Red, Thinking Bad: Color Bias in Vision Language Models

arXiv 2608.14286 ↗First posted 14 Aug 2026cs.CV cs.AI cs.CLcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 6 of its 42 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

6 of 42 references.

References · 6

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  3. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  4. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  5. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 19
  6. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Modelscited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  2. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  3. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
  4. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
  5. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  6. 2026Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm
  7. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  8. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models