# Don't Look at the Numbers: Visual Anchoring Bias and Layer-wise Representation in VLMs

arXiv: https://arxiv.org/abs/2605.11218 · HTML: https://mlsanity.com/paper/2605.11218/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 8 of 8)

- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Anchoring Bias in Large Language Models: An Experimental Study](https://mlsanity.com/paper/2412.06593/) (2024)
- [Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image](https://mlsanity.com/paper/2402.14899/) (2024)
- [Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation](https://mlsanity.com/paper/2511.05923/) (2025)
- [Vision Function Layer in Multimodal LLMs](https://mlsanity.com/paper/2509.24791/) (2025)
- [Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models](https://mlsanity.com/paper/2503.11519/) (2025)
- [Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP](https://mlsanity.com/paper/2508.20570/) (2025)
- [Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models](https://mlsanity.com/paper/2602.22918/) (2026)

## Abstract (arXiv)

Embedded numeric anchors on images systematically bias Vision-Language Model quality judgments across six VLMs from five architectural families (ANOVA eta^2 = 0.18-0.77, all p < 0.001). Anchor effects are 2.5x larger than severe image quality degradation, confirming bias is not reducible to visual changes. Layer-wise probing reveals consistent dissociation: layers where anchor classification saturates (L12-L34) are suboptimal for quality prediction, with optimal layers deeper (R^2 = 0.69-0.91). Fusion analysis identifies architecture-dependent integration -- instant fusion at L1-L2 in two models versus partial or no fusion in three others. These results establish a causal account of visual anchoring bias, linking behavioral susceptibility to representation dynamics.