mlsanityData to 14 Oct 2026

HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

arXiv 2403.00425 ↗First posted 1 Mar 2024cs.CV cs.AI cs.LGcited by 174

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

While large vision-language models (LVLMs) have demonstrated impressive capabilities in interpreting multi-modal contexts, they invariably suffer from object hallucinations (OH). We introduce HALC, a novel decoding algorithm designed to mitigate OH in LVLMs. HALC leverages distinct fine-grained optimal visual information in vision-language tasks and operates on both local and global contexts simultaneously. Specifically, HALC integrates a robust auto-focal grounding mechanism (locally) to correct hallucinated tokens on the fly, and a specialized beam search algorithm (globally) to significantly reduce OH while preserving text generation quality. Additionally, HALC can be integrated into any LVLMs as a plug-and-play module without extra training. Extensive experimental studies demonstrate the effectiveness of HALC in reducing OH, outperforming state-of-the-arts across four benchmarks.

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 49 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

41 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  3. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges6 Nov
  4. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
  9. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
20 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  5. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  7. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  8. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  9. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  10. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  11. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  12. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  13. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 94
  14. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  15. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  16. 2021On the Limitations of Multimodal VAEscited by 38
  17. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 29
  18. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  19. 2019Leveraging Sentence Similarity in Natural Language Generation: Improving Beam Search using Range Votingcited by 15
  20. 2023Multimodal Guidance Network for Missing-Modality Inference in Content Moderationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 73; 2026: 69.

174 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  3. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  4. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  5. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  6. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  7. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  8. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  9. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  10. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  11. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 52
  12. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
38 more
  1. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 48
  2. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
  3. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  4. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  5. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  6. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 38
  7. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  8. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  9. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  10. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 29
  11. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  12. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  13. 2025Towards Agentic Recommender Systems in the Era of Multimodal Large Language Modelscited by 26
  14. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 24
  15. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 23
  16. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  17. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  18. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learningcited by 21
  19. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  20. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Modelscited by 20
  21. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  22. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  23. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decodingcited by 19
  24. 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Frameworkcited by 17
  25. 2025VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsificationcited by 17
  26. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMscited by 17
  27. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  28. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Modelscited by 16
  29. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  30. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  31. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 14
  32. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  33. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
  34. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Modelscited by 12
  35. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decodingcited by 12
  36. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  37. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  38. 2025MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMscited by 11

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  2. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  3. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)
  4. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  5. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  6. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  7. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  8. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance