mlsanityData to 14 Oct 2026

Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue

arXiv 2212.05765 ↗First posted 12 Dec 2022cs.CL cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Video-grounded Dialogue (VGD) aims to decode an answer sentence to a question regarding a given video and dialogue context. Despite the recent success of multi-modal reasoning to generate answer sentences, existing dialogue systems still suffer from a text hallucination problem, which denotes indiscriminate text-copying from input texts without an understanding of the question. This is due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts, thus the VGD system excessively relies on copying words from input texts by hoping those words to overlap with ground-truth texts. Hence, we design Text Hallucination Mitigating (THAM) framework, which incorporates Text Hallucination Regularization (THR) loss derived from the proposed information-theoretic text hallucination measurement approach. Applying THAM with current dialogue systems validates the effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows enhanced interpretability.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

25 of 38 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2016Layer Normalizationcited by 5,669
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  12. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
13 more
  1. 2019Audio-Visual Scene-Aware Dialogcited by 123
  2. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  3. 2022Invariant Grounding for Video Question Answeringcited by 68
  4. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  5. 2018End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Featurescited by 37
  6. 2021Structured Co-reference Graph Attention for Video-grounded Dialoguecited by 17
  7. 2018From FiLM to Video: Multi-turn Question Answering with Multi-modal Contextcited by 12
  8. 2020Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue Systemcited by 11
  9. 2021VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Taskscited by 11
  10. 2020DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generatorcited by 9
  11. 2019Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generationcited by 7
  12. 2020Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challengecited by 5
  13. 2019Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modelingcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 11; 2025: 5.

18 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  2. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  3. 2024Mitigating Large Language Model Hallucination with Faithful Finetuningcited by 24
  4. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  5. 2024FRAG: Frequency Adapting Group for Diffusion Video Editingcited by 15
  6. 2023HEAR: Hearing Enhanced Audio Response for Video-grounded Dialoguecited by 13
  7. 2024TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animationcited by 12
  8. 2024FACTOID: FACtual enTailment fOr hallucInation Detectioncited by 10
  9. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  10. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8
  11. 2024Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expressioncited by 6
  12. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 6
6 more
  1. 2024LightHouse: A Survey of AGI Hallucinationcited by 4
  2. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  3. 2024Multi-Modal Video Dialog State Tracking in the Wildcited by 3
  4. 2025Occlusion-robust Stylization for Drawing-based 3D Animationcited by 3
  5. 2023Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialogcited by 1
  6. 2025V2Dial: Unification of Video and Visual Dialog via Multimodal Expertscited by 0