mlsanityData to 14 Oct 2026

Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

arXiv 2501.15269 ↗First posted 25 Jan 2025cs.LG cs.CR cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Fusing visual understanding into language generation, Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications. Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content. In this work, we delve into the internal attention mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing the inherent vulnerabilities in the instruction-tuning process. We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance, posing a significant threat to critical downstream applications. Distinguished from previous adversarial methods that rely on fixed patterns, our approach generates dynamic, effective, and highly transferable visual adversarial inputs, without sacrificing the quality of model responses. Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our attack in compromising black-box MLLMs even with extensive mitigating mechanisms, as well as the promising results against cutting-edge commercial APIs, such as GPT-4o and Gemini 1.5. Our code is available at https://huggingface.co/RachelHGF/Mirage-in-the-Eyes.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 86 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

73 of 86 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  2. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  3. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  4. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  8. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  9. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  10. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  11. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
2 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  2. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  9. 2023Vision Transformers Need Registerscited by 871
  10. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  13. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  14. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  15. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  16. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  17. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  18. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  19. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  20. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  21. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  22. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 162
  23. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  24. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  25. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  26. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 92
  27. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 81
  28. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  29. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  30. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  31. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  32. 2023Robust Evaluation of Diffusion-Based Adversarial Purificationcited by 65
  33. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
  34. 2024Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attackscited by 38
  35. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  36. 2023DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purificationcited by 36
  37. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  38. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 59
  2. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  3. 2025Don't Trust Your Upstream: Exploiting LLM Multi-Agent System via Topology-Guided Adversarial Propagationcited by 5
  4. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 4
  5. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  6. 2026When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Modelscited by 4
  7. 2026Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learningcited by 2
  8. 2025Investigating Faithfulness in Large Audio Language Modelscited by 1
  9. 2026Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing preventioncited by 1
  10. 2025Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Modelscited by 0
  11. 2025FeatureFool: Zero-Query Fooling of Video Models via Feature Mapcited by 0
  12. 2026A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigationcited by 0
3 more
  1. 2026Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurementcited by 0
  2. 2026Babel: Jailbreaking Safety Attention via Obfuscation Distribution Optimized Samplingcited by 0
  3. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  2. 2026Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
  3. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  4. 2025Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMs
  5. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination
  6. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
  7. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  8. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model