# Prior Directions: Why GUI Grounding Gets Locked in the Past

arXiv: https://arxiv.org/abs/2607.26913 · HTML: https://mlsanity.com/paper/2607.26913/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 25 of 25)

- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [Words or Vision: Do Vision-Language Models Have Blind Faith in Text?](https://mlsanity.com/paper/2503.02199/) (2025)
- [Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection](https://mlsanity.com/paper/2412.13817/) (2024)
- [VLind-Bench: Measuring Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2406.08702/) (2024)
- [Probing Visual Language Priors in VLMs](https://mlsanity.com/paper/2501.00569/) (2024)
- [Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts](https://mlsanity.com/paper/2505.17127/) (2025)
- [What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation](https://mlsanity.com/paper/2406.16320/) (2024)
- [Finding Visual Task Vectors](https://mlsanity.com/paper/2404.05729/) (2024)
- [Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models](https://mlsanity.com/paper/2502.16033/) (2025)
- [How Do Vision-Language Models Process Conflicting Information Across Modalities?](https://mlsanity.com/paper/2507.01790/) (2025)
- [Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models](https://mlsanity.com/paper/2505.19509/) (2025)
- [When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models](https://mlsanity.com/paper/2507.13868/) (2025)
- [MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents](https://mlsanity.com/paper/2605.18652/) (2026)
- [Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models](https://mlsanity.com/paper/2602.21704/) (2026)
- [Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark](https://mlsanity.com/paper/2606.10400/) (2026)
- [Mechanisms of Prompt-Induced Hallucination in Vision-Language Models](https://mlsanity.com/paper/2601.05201/) (2026)
- [Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models](https://mlsanity.com/paper/2502.14888/) (2025)
- [Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2602.11824/) (2026)
- [Steering Vision-Language Models with Joint Sparse Autoencoders](https://mlsanity.com/paper/2606.25657/) (2026)
- [Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models](https://mlsanity.com/paper/2606.28273/) (2026)
- [When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs](https://mlsanity.com/paper/2604.21911/) (2026)
- [Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating](https://mlsanity.com/paper/2605.24024/) (2026)
- [When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models](https://mlsanity.com/paper/2605.08245/) (2026)

## Abstract (arXiv)

Vision-language models can remain anchored to stale verbal priors even when current visual evidence supports a different answer. We use this controlled revision failure to ask whether the subspace that best captures recurrent prior-induced representation change is also uniquely more behaviorally effective. Across four models, paired prior and reference states define Prior Directions, a compact low-rank geometry that, at rank 32, spans only 0.78% of the state dimensions yet captures 49.7% to 64.0% of held-out displacement energy. Matched correspondence improves held-out capture at every tested rank in all four models; across 20 permutation seeds, this advantage remains positive in all 15 available combinations of model and rank. Yet the geometric advantage does not translate into a comparable intervention advantage: matched and Pairing-Permuted edits restore 60/66 and 59/66 clean prior-induced failures, respectively. We trace this discrepancy to the geometry of the realized edits. Pairing-Permuted edits remain strongly aligned with the matched Prior Directions; their aligned component restores 59/66 cases, whereas an equally norm-matched orthogonal residual restores only 20/66. These results show that the construction that best identifies recurrent representation change need not be uniquely more behaviorally effective, distinguishing global subspace identification from the geometry actually used by an intervention.