mlsanityData to 7 Oct 2026

Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination

arXiv 2403.14401 ↗First posted 21 Mar 2024cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) demonstrate remarkable success across various vision-language tasks. However, they suffer from visual hallucination, where the generated responses diverge from the provided image. Are MLLMs oblivious to the accurate visual cues when they hallucinate? Our investigation reveals that the visual branch may equally advocate both accurate and erroneous content. To address this issue, we propose Pensieve, a training-free method that leverages the analogous visual hallucinations, which are induced by images sharing common semantic and appearance characteristics, to mitigate hallucination. Specifically, Pensieve enables MLLMs to retrospect relevant images as references and compare their visual content with the test image via confidence score subtraction. Moreover, our paradigm balances the effects of addressing errors from both the visual and textual branches by adaptively scaling the subtracted scores. Experiments on Whoops, LLaVA Bench, POPE, and MME demonstrate the efficacy of Pensieve in mitigating visual hallucination, surpassing other advanced decoding strategies. Pensieve also aids MLLMs in identifying visual details and enhance the specificity of generated image descriptions.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 80 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

63 of 80 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  2. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset17 Mar
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  9. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  10. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  11. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
2 more
  1. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  2. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
37 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  15. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  16. 2018Object Hallucination in Image Captioningcited by 661
  17. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  18. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  19. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  20. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  21. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  22. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  23. 2023How Language Model Hallucinations Can Snowballcited by 310
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  25. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  26. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  27. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  28. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  29. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  30. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  31. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  32. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  33. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  34. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuningcited by 73
  35. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  36. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 28
  37. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  2. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  4. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  5. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  6. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decodingcited by 8
  7. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8
  8. 2024RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Modelscited by 7
  9. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Modelscited by 5
  10. 2025Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMscited by 5
  11. 2025Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Modelscited by 3
  12. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Modelcited by 2
2 more
  1. 2026YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Modelscited by 0
  2. 2026OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  2. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  3. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  4. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models
  5. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
  6. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  7. 2025SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
  8. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating