mlsanityData to 14 Oct 2026

Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

arXiv 2604.12119 ↗First posted 13 Apr 2026cs.CV cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap. Prompt interventions support this mechanism: neutral alias prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and dataset available at https://maveryn.github.io/vlm-fix/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 44 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

29 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Vision Language Models are Biased29 May
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  3. 2024VisMin: Visual Minimal-Change Understanding23 Jul
  4. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models13 Jun
  5. 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense30 Oct

Other references · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  9. 2025OpenAI GPT-5 System Cardcited by 989
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
12 more
  1. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  2. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  3. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
  4. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  5. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  6. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  7. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  8. 2024A Vision Check-up for Language Modelscited by 36
  9. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 31
  10. 2022CPL: Counterfactual Prompt Learning for Vision and Language Modelscited by 18
  11. 2023Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Datasetcited by 17
  12. 2024Causal-Guided Active Learning for Debiasing Large Language Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mind the Gap: Diagnosing Constraint Discovery Failures in Text-in-Image Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-Time Consistency in Vision Language Models
  2. 2026PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models
  3. 2026Semantic Misalignment in Vision-Language Models under Perceptual Degradation
  4. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective
  5. 2025Token-Level Inference-Time Alignment for Vision-Language Models
  6. 2026Stable Language Guidance for Vision-Language-Action Models
  7. 2026Evaluating Self-Correcting Vision Agents Through Quantitative and Qualitative Metrics
  8. 2026Test-Time Training for Modality Order Consistency in Vision-Language Models