mlsanityData to 14 Oct 2026

Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMs

arXiv 2504.09456 ↗First posted 13 Apr 2025cs.AI cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Multimodal Models (LMMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their vulnerability to user gaslighting-the deliberate use of misleading or contradictory inputs-raises critical concerns about their reliability in real-world applications. In this paper, we address the novel and challenging issue of mitigating the negative impact of negation-based gaslighting on LMMs, where deceptive user statements lead to significant drops in model accuracy. Specifically, we introduce GasEraser, a training-free approach that reallocates attention weights from misleading textual tokens to semantically salient visual regions. By suppressing the influence of "attention sink" tokens and enhancing focus on visually grounded cues, GasEraser significantly improves LMM robustness without requiring retraining or additional supervision. Extensive experimental results demonstrate that GasEraser is effective across several leading open-source LMMs on the GaslightingBench. Notably, for LLaVA-v1.5-7B, GasEraser reduces the misguidance rate by 48.2%, demonstrating its potential for more trustworthy LMMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 35 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

18 of 35 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  11. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  12. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
6 more
  1. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  2. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  3. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  4. 2025Vision-Language Models Do Not Understand Negationcited by 72
  5. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 56
  6. 2025Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibrationcited by 12
  2. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  3. 2025Benchmarking Gaslighting Negation Attacks Against Reasoning Modelscited by 6
  4. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  5. 2026Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Modelscited by 1
  6. 2025Benchmarking Gaslighting Attacks Against Speech Large Language Modelscited by 0
  7. 2026Disentangling Semantic Attention from Structural Bias in the Attention Manifoldcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
  2. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
  3. 2026Visual Attention Faithfulness in Vision-Language Models is Heterogeneous
  4. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration
  5. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
  6. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  7. 2024The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
  8. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering