mlsanityData to 14 Oct 2026

Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm

arXiv 2610.12650 ↗First posted 8 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs also fits: VLM benchmarks and visual reasoning

Abstract (arXiv)

When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image. But to act means to choose: saying one thing means choosing to respond not say another. This becomes an issue when the image contains multiple features that equally relate to the question at hand: how would VLMs choose to respond? This problem is precisely studied by the classical Stroop task in psychology: among a set of deliberately designed images whose text and color conflicts (e.g., textcolorblueRED), participants are instructed to report the color. We adapt this psychological paradigm to a benchmark and prompted a range of contrastive and generative VLMs on their behavior. Across models, we observe a persistent bias toward the feature of text in Stroop images: when text and color disagree, models preferentially generate responses centered to the text rather than color. We then opened the black box and examined the source of this bias and why certain features influences the model's behavior more than others. In the embedding space, we found that textual cues are represented more strongly and separably than color cues. This representational asymmetry predicts intervention success: steering models toward text-related directions is substantially easier than steering them toward color-related directions. Using a classical psychological paradigm, we demonstrate, at both behavioral and representational levels, that distinct features within the same image can exert unequal influence on VLM behavior. Our findings suggest that VLMs assign different degrees of saliency to different features within the same image, highlighting the need to understand how cue-specific saliency shapes their performance in widely used downstream applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 35 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

13 of 35 references.

References · 13

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  7. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  8. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  9. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48
  10. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 10
  11. 2025Color in Visual-Language Models: CLIP deficienciescited by 4
  12. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 4
1 more
  1. 2025Concept-Guided Interpretability via Neural Chunkingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Red, Thinking Bad: Color Bias in Vision Language Models
  2. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  3. 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Models
  4. 2025The in-context inductive biases of vision-language models differ across modalities
  5. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
  6. 2026Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
  7. 2026Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models
  8. 2026When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs