mlsanityData to 14 Oct 2026

OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

arXiv 2311.17911 ↗First posted 29 Nov 2023cs.CVcited by 478

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucination, posed as a pervasive challenge of multi-modal large language models (MLLMs), has significantly impeded their real-world usage that demands precise judgment. Existing methods mitigate this issue with either training with specific designed data or inferencing with external knowledge from other sources, incurring inevitable additional costs. In this paper, we present OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a Retrospection-Allocation strategy, serving as a nearly free lunch to alleviate the hallucination issue without additional data, knowledge, or training. Our approach begins with an interesting observation that, most hallucinations are closely tied to the knowledge aggregation patterns manifested in the self-attention matrix, i.e., MLLMs tend to generate new tokens by focusing on a few summary tokens, but not all the previous tokens. Such partial over-trust inclination results in the neglecting of image tokens and describes the image content with hallucination. Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue, along with a rollback strategy that retrospects the presence of summary tokens in the previously generated tokens, and re-allocate the token selection if necessary. With extensive experiments, OPERA shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality. Our code is available at: https://github.com/shikiw/OPERA.

Key facts. Cited by 478 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 51 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

42 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  2. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
26 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  7. 2018Hierarchical Neural Story Generationcited by 1,193
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  12. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  13. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  14. 2018Object Hallucination in Image Captioningcited by 669
  15. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  16. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  17. 2023How Language Model Hallucinations Can Snowballcited by 311
  18. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  19. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  20. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  21. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  22. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  23. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  24. 2023Diversity-Aware Meta Visual Promptingcited by 58
  25. 2022Shape-invariant 3D Adversarial Point Cloudscited by 50
  26. 2021Initiative Defense against Facial Manipulationcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 114; 2025: 172; 2026: 191.

478 papers.

Sort:

By area:

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  3. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  4. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  5. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  7. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  8. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  9. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  10. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 157
  11. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  12. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
38 more
  1. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  2. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  3. 2025Latent Visual Reasoningcited by 104
  4. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
  5. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 94
  6. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  7. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  8. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  9. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  10. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  11. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  12. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
  13. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  14. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  15. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  16. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  17. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  18. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  19. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 65
  20. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 65
  21. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 60
  22. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  23. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  24. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
  25. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  26. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  27. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 52
  28. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  29. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  30. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  31. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
  32. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 48
  33. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Modelscited by 47
  34. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
  35. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 47
  36. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46
  37. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  38. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45

50 most cited of 478 All 478 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  2. 2025Theoretical Foundations and Mitigation of Hallucination in Large Language Models
  3. 2026Predictive Coding and Information Bottleneck for Hallucination Detection in Large Language Models
  4. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
  5. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  6. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
  7. 2026KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation
  8. 2025HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models