mlsanityData to 7 Oct 2026

Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Models

arXiv 2412.04939 ↗First posted 6 Dec 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have garnered significant attention recently and demonstrate outstanding capabilities in various tasks such as OCR, VQA, captioning, textitetc. However, hallucination remains a persistent issue. While numerous methods have been proposed to mitigate hallucinations, achieving notable improvements, these methods primarily focus on mitigating hallucinations about object/noun-related concepts. Verb concepts, crucial for understanding human actions, have been largely overlooked. In this paper, to the best of our knowledge, we are the first to investigate the verb hallucination phenomenon of MLLMs from various perspectives. Our findings reveal that most state-of-the-art MLLMs suffer from severe verb hallucination. To assess the effectiveness of existing mitigation methods for object concept hallucination on verb hallucination, we evaluated these methods and found that they do not effectively address verb hallucination. To address this issue, we propose a novel rich verb knowledge-based tuning method to mitigate verb hallucination. The experiment results demonstrate that our method significantly reduces hallucinations related to verbs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 70 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

41 of 70 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning22 Jul
  2. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
23 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  7. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  12. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  13. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  14. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  15. 2021Partial success in closing the gap between human and machine visioncited by 185
  16. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  17. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  18. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  19. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 79
  20. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  21. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  22. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  23. 2024The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
  3. 2024Hallucination of Multimodal Large Language Models: A Survey
  4. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models
  5. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
  6. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models
  7. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models
  8. 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Models