mlsanityData to 7 Oct 2026

Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

arXiv 2410.03577 ↗First posted 4 Oct 2024cs.CVcited by 72

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite their impressive capabilities, multimodal large language models (MLLMs) are prone to hallucinations, i.e., the generated content that is nonsensical or unfaithful to input sources. Unlike in LLMs, hallucinations in MLLMs often stem from the sensitivity of text decoder to visual tokens, leading to a phenomenon akin to "amnesia" about visual information. To address this issue, we propose MemVR, a novel decoding paradigm inspired by common cognition: when the memory of an image seen the moment before is forgotten, people will look at it again for factual answers. Following this principle, we treat visual tokens as supplementary evidence, re-injecting them into the MLLM through Feed Forward Network (FFN) as "key-value memory" at the middle trigger layer. This "look-twice" mechanism occurs when the model exhibits high uncertainty during inference, effectively enhancing factual alignment. Comprehensive experimental evaluations demonstrate that MemVR significantly mitigates hallucination across various MLLMs and excels in general benchmarks without incurring additional time overhead. The implementation is available from https://github.com/1zhou-Wang/MemVR

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 78 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

60 of 78 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning30 Aug
  2. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models25 Aug
  3. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models18 Aug
  4. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  5. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation1 Aug
  6. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  7. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  8. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  9. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  10. 2024Visual Hallucinations of Multi-modal Large Language Models22 Feb
  11. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
8 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  3. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  6. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
28 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  8. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  9. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  10. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2023Generating Images with Multimodal Language Modelscited by 259
  13. 2019Depth-Adaptive Transformercited by 227
  14. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  15. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  16. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  17. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  18. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  19. 2020Evolving Normalization-Activation Layerscited by 45
  20. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  21. 2024IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologuescited by 40
  22. 2021On the Limitations of Multimodal VAEscited by 37
  23. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  24. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  25. 2024The Evolution of Multimodal Model Architecturescited by 21
  26. 2024Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuningcited by 20
  27. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  28. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethicscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 22; 2026: 45.

72 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  5. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  6. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  7. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  8. 2025Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Spacecited by 25
  9. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  10. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  11. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decodingcited by 14
  12. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
38 more
  1. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 11
  2. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  3. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  4. 2026Hallucination Begins Where Saliency Dropscited by 8
  5. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
  6. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  7. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6
  8. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  9. 2026Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimizationcited by 6
  10. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  11. 2026UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Modelscited by 5
  12. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  13. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  14. 2025Boosting Reasoning in Large Multimodal Models via Activation Replaycited by 4
  15. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  16. 2024Trusted Mamba Contrastive Network for Multi-View Clusteringcited by 3
  17. 2025V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Interventioncited by 3
  18. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  19. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  20. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 3
  21. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Modelscited by 2
  22. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 2
  23. 2026Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoningcited by 2
  24. 2026WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMscited by 2
  25. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 2
  26. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMcited by 2
  27. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  28. 2026Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinationscited by 2
  29. 2026Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theorycited by 2
  30. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  31. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  32. 2026Visual Access Boundaries in Vision-Language Model Reasoningcited by 2
  33. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  34. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  35. 2026GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinationscited by 1
  36. 2026Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Modelscited by 1
  37. 2026Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Timecited by 1
  38. 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMscited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  4. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination
  5. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  6. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
  7. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  8. 2026Hallucination-aware intermediate representation edit in large vision-language models