mlsanityData to 7 Oct 2026

Prior Directions: Why GUI Grounding Gets Locked in the Past

arXiv 2607.26913 ↗First posted 29 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models can remain anchored to stale verbal priors even when current visual evidence supports a different answer. We use this controlled revision failure to ask whether the subspace that best captures recurrent prior-induced representation change is also uniquely more behaviorally effective. Across four models, paired prior and reference states define Prior Directions, a compact low-rank geometry that, at rank 32, spans only 0.78% of the state dimensions yet captures 49.7% to 64.0% of held-out displacement energy. Matched correspondence improves held-out capture at every tested rank in all four models; across 20 permutation seeds, this advantage remains positive in all 15 available combinations of model and rank. Yet the geometric advantage does not translate into a comparable intervention advantage: matched and Pairing-Permuted edits restore 60/66 and 59/66 clean prior-induced failures, respectively. We trace this discrepancy to the geometry of the realized edits. Pairing-Permuted edits remain strongly aligned with the matched Prior Directions; their aligned component restores 59/66 cases, whereas an equally norm-matched orthogonal residual restores only 20/66. These results show that the construction that best identifies recurrent representation change need not be uniquely more behaviorally effective, distinguishing global subspace identification from the geometry actually used by an intervention.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 29 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

25 of 29 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models26 Jun
  2. 2026Steering Vision-Language Models with Joint Sparse Autoencoders24 Jun
  3. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark9 Jun
  4. 2026When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models7 May
  5. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models18 Jul
  6. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?2 Jul
  7. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May
  8. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  9. 2024Probing Visual Language Priors in VLMs31 Dec
  10. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation24 Jun
  11. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models13 Jun

Other references · 14

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  2. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  3. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  4. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  5. 2024Finding Visual Task Vectorscited by 24
  6. 2025Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Modelscited by 24
  7. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 18
  8. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  9. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Modelscited by 6
  10. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  11. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Modelscited by 3
  12. 2026Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Modelscited by 3
2 more
  1. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  2. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gatingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models
  2. 2026GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
  3. 2026Knowing Isn't Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?
  4. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
  5. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
  6. 2026Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
  7. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
  8. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning