mlsanityData to 7 Oct 2026

Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

arXiv 2606.27596 ↗First posted 25 Jun 2026cs.CV cs.AIcited by 3

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) exhibit sophisticated reasoning but remain susceptible to object hallucination. Deviating from the prevailing attention intensity assumption, we reveal a deeper dynamic structural misalignment: hallucination is triggered at decision-critical steps where specific attention heads, acting as risky mediators, decouple from visual evidence to lock onto language priors. This establishes a pathological shortcut that bypasses visual grounding. To dismantle this, we propose Fox (Faithfulness and Observational-flow via eXpression-rectification), a training-free inference-time framework. Fox diagnoses structural misalignment using a visual attention entropy probe to localize risky mediators unsupervisedly. We then execute a targeted causal intervention via numerical logit saturation to physically sever the shortcut path. Finally, a conflict-gated cooperative decoding strategy reconciles interventional faithfulness with observational fluency. Extensive experiments demonstrate that Fox achieves SOTA performance, outperforming SID by 29.1% while preserving linguistic richness. Code is available at https://github.com/Cc2021start/Fox.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 30 of its 41 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

30 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding29 Sep

Other references · 29

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  12. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
17 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  5. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  6. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  7. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  8. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  9. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  10. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  11. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  12. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 10
  13. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMscited by 8
  14. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flowcited by 7
  15. 2025Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentencescited by 5
  16. 2024Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysiscited by 4
  17. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learningcited by 5
  2. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Modelscited by 0
  3. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
  2. 2026Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
  3. 2024Mitigating Object Hallucination via Concentric Causal Attention
  4. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models
  5. 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
  6. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
  7. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
  8. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration