mlsanityData to 7 Oct 2026

Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites

arXiv 2312.01701 ↗First posted 4 Dec 2023cs.CV cs.CLcited by 61

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large language models (LLMs) have shown remarkable performance in natural language processing (NLP) tasks. To comprehend and execute diverse human instructions over image data, instruction-tuned large vision-language models (LVLMs) have been introduced. However, LVLMs may suffer from different types of object hallucinations. Nevertheless, LVLMs are evaluated for coarse-grained object hallucinations only (i.e., generated objects non-existent in the input image). The fine-grained object attributes and behaviors non-existent in the image may still be generated but not measured by the current evaluation methods. In this paper, we thus focus on reducing fine-grained hallucinations of LVLMs. We propose textitReCaption, a framework that consists of two components: rewriting captions using ChatGPT and fine-tuning the instruction-tuned LVLMs on the rewritten captions. We also propose a fine-grained probing-based evaluation method named textitFine-Grained Object Hallucination Evaluation (textitFGHE). Our experiment results demonstrate that ReCaption effectively reduces fine-grained object hallucination for different LVLM options and improves their text generation quality. The code can be found at https://github.com/Anonymousanoy/FOHE.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 39 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

27 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  2. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  3. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  9. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
12 more
  1. 2018nocaps: novel object captioning at scalecited by 444
  2. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  3. 2023Improving CLIP Training with Language Rewritescited by 216
  4. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  5. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  6. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  7. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  8. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  9. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  10. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  11. 2020A Controllable Model of Grounded Response Generationcited by 49
  12. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 21; 2026: 11.

61 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  3. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  5. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  6. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  7. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  8. 2024Building Guardrails for Large Language Modelscited by 61
  9. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  10. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  11. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 30
  12. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
38 more
  1. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  2. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 27
  3. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27
  4. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  5. 2024CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generationcited by 25
  6. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)cited by 23
  7. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  8. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  9. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  10. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmarkcited by 18
  11. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  12. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  13. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  14. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Informationcited by 11
  15. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  16. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
  17. 2025Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Modelscited by 9
  18. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decodingcited by 8
  19. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 7
  20. 2024Robustness of Structured Data Extraction from In-plane Rotated Documents using Multi-Modal Large Language Models (LLM)cited by 7
  21. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
  22. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  23. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  24. 2026Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volumecited by 7
  25. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMscited by 5
  26. 2024LightHouse: A Survey of AGI Hallucinationcited by 4
  27. 2024What if...?: Thinking Counterfactual Keywords Helps to Mitigate Hallucination in Large Multi-modal Modelscited by 4
  28. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  29. 2025Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillationcited by 4
  30. 2024Mitigating Multilingual Hallucination in Large Vision-Language Modelscited by 3
  31. 2025See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMscited by 3
  32. 2024Quantity Matters: Towards Assessing and Mitigating Number Hallucination in Large Vision-Language Modelscited by 2
  33. 2025PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoningcited by 2
  34. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  35. 2026SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineeringcited by 2
  36. 2025GE-Chat: A Graph Enhanced RAG Framework for Evidential Response Generation of LLMscited by 1
  37. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Imagescited by 1
  38. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating Object Hallucination in Large Vision-Language Models
  2. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models
  3. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  4. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
  5. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  7. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  8. 2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models