mlsanityData to 7 Oct 2026

ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models

arXiv 2403.16167 ↗First posted 24 Mar 2024cs.CV cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucinations in vision-language models pose a significant challenge to their reliability, particularly in the generation of long captions. Current methods fall short of accurately identifying and mitigating these hallucinations. To address this issue, we introduce ESREAL, a novel unsupervised learning framework designed to suppress the generation of hallucinations through accurate localization and penalization of hallucinated tokens. Initially, ESREAL creates a reconstructed image based on the generated caption and aligns its corresponding regions with those of the original image. This semantic reconstruction aids in identifying both the presence and type of token-level hallucinations within the generated caption. Subsequently, ESREAL computes token-level hallucination scores by assessing the semantic similarity of aligned regions based on the type of hallucination. Finally, ESREAL employs a proximal policy optimization algorithm, where it selectively penalizes hallucinated tokens according to their token-level hallucination scores. Our framework notably reduces hallucinations in LLaVA, InstructBLIP, and mPLUG-Owl2 by 32.81%, 27.08%, and 7.46% on the CHAIR metric. This improvement is achieved solely through signals derived from the image itself, without the need for any image-text pairs.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

33 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  3. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  4. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  5. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
15 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2023Adversarial Diffusion Distillationcited by 741
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  13. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  14. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  15. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 11.

12 papers.

  1. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  2. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  3. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  4. 2025HalLoc: Token-level Localization of Hallucinations for Vision Language Modelscited by 9
  5. 2025WeatherPrompt: Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localizationcited by 7
  6. 2025ZINA: Multimodal Fine-grained Hallucination Detection and Editingcited by 5
  7. 2025Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observationscited by 4
  8. 2024CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interactioncited by 3
  9. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  10. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  11. 2025Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -cited by 1
  12. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
  2. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors
  3. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  5. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  6. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  7. 2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models
  8. 2025Suppressing VLM Hallucinations with Spectral Representation Filtering