mlsanityData to 7 Oct 2026

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

arXiv 2606.07647 ↗First posted 2 Jun 2026cs.CV cs.CL cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

39 of 46 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs26 May
  2. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding22 May
  3. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models9 Apr
  4. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  5. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage20 Dec
  6. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  7. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  8. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  9. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  10. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  12. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
4 more
  1. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 23

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2018Object Hallucination in Image Captioningcited by 661
11 more
  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  2. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  3. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  4. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  5. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  6. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  7. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  8. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  9. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  10. 2025Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generationcited by 10
  11. 2025CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Trainingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
  2. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs
  3. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
  4. 2026Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation
  5. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
  6. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
  7. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  8. 2026Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction