mlsanityData to 14 Oct 2026

From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models

arXiv 2410.06795 ↗First posted 9 Oct 2024cs.CL cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucinations in large vision-language models (LVLMs) are a significant challenge, i.e., generating objects that are not presented in the visual input, which impairs their reliability. Recent studies often attribute hallucinations to a lack of understanding of visual input, yet ignore a more fundamental issue: the model's inability to effectively extract or decouple visual features. In this paper, we revisit the hallucinations in LVLMs from an architectural perspective, investigating whether the primary cause lies in the visual encoder (feature extraction) or the modal alignment module (feature decoupling). Motivated by our findings on the preliminary investigation, we propose a novel tuning strategy, PATCH, to mitigate hallucinations in LVLMs. This plug-and-play method can be integrated into various LVLMs, utilizing adaptive virtual tokens to extract object features from bounding boxes, thereby addressing hallucinations caused by insufficient decoupling of visual features. PATCH achieves state-of-the-art performance on multiple multi-modal hallucination datasets. We hope this approach provides researchers with deeper insights into the underlying causes of hallucinations in LVLMs, fostering further advancements and innovation in this field.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 49 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

40 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination21 Mar
  2. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  5. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  6. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 32

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
20 more
  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  7. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  8. 2023Segment Everything Everywhere All at Oncecited by 548
  9. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  10. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  11. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  12. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  13. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  14. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  15. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  16. 2020A Survey on Instance Segmentation: State of the artcited by 85
  17. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  18. 2024Exploring the Robustness of Decision-Level Through Adversarial Attacks on LLM-Based Embodied Modelscited by 33
  19. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  20. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Hallucination Begins Where Saliency Dropscited by 9
  2. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinationscited by 3
  3. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  4. 2025What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Promptcited by 1
  5. 2026DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2024A Survey on Hallucination in Large Vision-Language Models
  4. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  5. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
  7. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention
  8. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models