The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning also fits: VLM benchmarks and visual reasoning
Abstract (arXiv)
Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve predictions. First, we apply the same visual-residual edit at successive decoder layers, matching its RMS strength across depth. Across six checkpoints from three model families, we observe a persistent drop in intervention effect and define its operational onset for this edit as the Evidence Anchoring Boundary (EAB). With boundaries fixed on calibration queries, held-out effects on disjoint evaluation queries before EAB are 13.78×--26.10× larger than those after it. Comparison with intermediate readouts shows that the edit can lose leverage before answer preferences stabilize. Second, we compare counterfactual subtraction across presence, attribute, and relation questions. Subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer. Based on these observations, we develop calibrated selective routing, which preserves the original prediction unless calibration supports an intervention for the question type. It improves full-set attribute accuracy by 1.56--1.81 percentage points across three checkpoints, while universal subtraction is neutral or harmful.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 28 references that are in the map. First posted on arXiv 9 Oct 2026.
Builds on
24 of 28 references.
Closest ancestors · 3
Same direction, newest first
- 2025Vision Function Layer in Multimodal LLMs29 Sep
- 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs24 Feb
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
Other references · 21
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
- 2018Object Hallucination in Image Captioningcited by 669
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
- 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
- 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
- 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
9 more
- 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
- 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
- 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
- 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
- 2026Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibrationcited by 1
- 2026VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answeringcited by 1
- 2026CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMscited by 1
- 2026What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustnesscited by 1
- 2026YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigations
- 2026AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs
- 2026When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models
- 2026HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide
- 2026Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models
- 2026Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs
- 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
- 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act