mlsanityData to 14 Oct 2026

Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models

arXiv 2508.01678 ↗First posted 3 Aug 2025cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-Language Models (VLMs) often suffer from hallucination, partly due to challenges in aligning multimodal information. We propose Prompt-in-Image, a simple method that embeds textual instructions directly into images. This removes the need for separate text inputs and forces the model to process all content through the visual channel. We evaluate this method on three popular open-source VLMs: Qwen2.5-VL, LLaVA-1.5, and InstructBLIP. The results reveal sharp differences. Prompt-in-Image improves Qwen2.5-VL's performance, increasing POPE accuracy by 4.1 percent (from 80.2 percent to 84.3 percent) and also reducing hallucination rates on MS-COCO. In contrast, LLaVA-1.5 and InstructBLIP experience a severe performance drop, with accuracy falling from around 84 percent to near-random levels. Through detailed analysis, we found that CLIP-based encoders in LLaVA and InstructBLIP exhibit excessive attention bias toward embedded text regions, disrupting visual understanding. In contrast, Qwen's vision encoder handles text-embedded images robustly. Crucially, Prompt-in-Image reduces Qwen's modality gap, enhancing cross-modal alignment by unifying information processing through a single modality.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 27 references that are in the map. First posted on arXiv 3 Aug 2025.

Builds on

19 of 27 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?14 Apr
  2. 2024DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models27 Nov
  3. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  6. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  9. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 10

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2023Vision Transformers Need Registerscited by 871
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2018Object Hallucination in Image Captioningcited by 669
  6. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  7. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  8. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  9. 2025Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learningcited by 8
  10. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  2. 2026Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMscited by 7
  3. 2026Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseasescited by 0
  4. 2026Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  4. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
  5. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Models
  6. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  7. 2025Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models
  8. 2024Multi-Modal Hallucination Control by Visual Information Grounding