mlsanityData to 7 Oct 2026

Can Hallucination Correction Improve Video-Language Alignment?

arXiv 2502.15079 ↗First posted 20 Feb 2025cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models often generate hallucinated content that is not grounded in its visual inputs. While prior work focuses on mitigating hallucinations, we instead explore leveraging hallucination correction as a training objective to improve video-language alignment. We introduce HACA, a self-training framework learning to correct hallucinations in descriptions that do not align with the video content. By identifying and correcting inconsistencies, HACA enhances the model's ability to align video and textual representations for spatio-temporal reasoning. Our experimental results show consistent gains in video-caption binding and text-to-video retrieval tasks, demonstrating that hallucination correction-inspired tasks serve as an effective strategy for improving vision and language alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 53 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

37 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 34

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
22 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  4. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  5. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  6. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  7. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  8. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  10. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  11. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  12. 2018Localizing Moments in Video with Temporal Languagecited by 83
  13. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  14. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  15. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  16. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  17. 2019Only Time Can Tell: Discovering Temporal Data for Temporal Modelingcited by 66
  18. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  19. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  20. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  21. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  22. 2024Successfully Guiding Humans with Imperfect Instructions by Highlighting Potential Errors and Suggesting Correctionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2025See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs
  4. 2025ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations
  5. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
  6. 2026Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models
  7. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding
  8. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence