mlsanityData to 7 Oct 2026

Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

arXiv 2501.19164 ↗First posted 31 Jan 2025cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 59 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

45 of 59 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  2. 2024BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models18 Jul
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models24 Mar
  5. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  7. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  11. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
4 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  11. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
17 more
  1. 2024What matters when building vision-language models?cited by 321
  2. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  3. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  4. 2023Language Modeling Is Compressioncited by 225
  5. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  6. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  7. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  8. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  9. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  10. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  11. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 89
  12. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  13. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  14. 2024SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Modelscited by 39
  15. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  16. 2024Novel Object Synthesis via Adaptive Text-Image Harmonycited by 12
  17. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  2. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  3. 2025ERC-SVD: Error-Controlled SVD for Large Language Model Compressioncited by 6
  4. 2026PASA: A Principled Embedding-Space Watermarking Approach for LLM-Generated Text under Semantic-Invariant Attackscited by 4
  5. 2025HCCM: Hierarchical Cross-Granularity Contrastive and Matching Learning for Natural Language-Guided Dronescited by 2
  6. 2026ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Bettercited by 2
  7. 2026Unified Hallucination Fuzzing for Multimodal Large Language Modelscited by 1
  8. 2025MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMscited by 0
  9. 2026MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Partscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMs
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  4. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  5. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  6. 2026Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation
  7. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens