# Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue

arXiv: https://arxiv.org/abs/2212.05765 · HTML: https://mlsanity.com/paper/2212.05765/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2206.02349/) (2022)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features](https://mlsanity.com/paper/1806.08409/) (2018)
- [Structured Co-reference Graph Attention for Video-grounded Dialogue](https://mlsanity.com/paper/2103.13361/) (2021)
- [From FiLM to Video: Multi-turn Question Answering with Multi-modal Context](https://mlsanity.com/paper/1812.07023/) (2018)
- [Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System](https://mlsanity.com/paper/2001.06206/) (2020)
- [VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks](https://mlsanity.com/paper/2104.07921/) (2021)
- [DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generator](https://mlsanity.com/paper/2004.08299/) (2020)
- [Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation](https://mlsanity.com/paper/1908.08191/) (2019)
- [Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge](https://mlsanity.com/paper/2002.10695/) (2020)
- [Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling](https://mlsanity.com/paper/1908.05067/) (2019)

## Built on by (most cited first, 18 of 18)

- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2407.16574/) (2024)
- [Mitigating Large Language Model Hallucination with Faithful Finetuning](https://mlsanity.com/paper/2406.11267/) (2024)
- [An Audit on the Perspectives and Challenges of Hallucinations in NLP](https://mlsanity.com/paper/2404.07461/) (2024)
- [FRAG: Frequency Adapting Group for Diffusion Video Editing](https://mlsanity.com/paper/2406.06044/) (2024)
- [HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue](https://mlsanity.com/paper/2312.09736/) (2023)
- [TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation](https://mlsanity.com/paper/2410.24037/) (2024)
- [FACTOID: FACtual enTailment fOr hallucInation Detection](https://mlsanity.com/paper/2403.19113/) (2024)
- [ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization](https://mlsanity.com/paper/2506.08712/) (2025)
- [BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2408.05926/) (2024)
- [Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expression](https://mlsanity.com/paper/2405.00301/) (2024)
- [Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models](https://mlsanity.com/paper/2507.04976/) (2025)
- [LightHouse: A Survey of AGI Hallucination](https://mlsanity.com/paper/2401.06792/) (2024)
- [Policy Learning from Large Vision-Language Model Feedback without Reward Modeling](https://mlsanity.com/paper/2507.23391/) (2025)
- [Multi-Modal Video Dialog State Tracking in the Wild](https://mlsanity.com/paper/2407.02218/) (2024)
- [Occlusion-robust Stylization for Drawing-based 3D Animation](https://mlsanity.com/paper/2508.00398/) (2025)
- [Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog](https://mlsanity.com/paper/2310.07259/) (2023)
- [V2Dial: Unification of Video and Visual Dialog via Multimodal Experts](https://mlsanity.com/paper/2503.02063/) (2025)

## Abstract (arXiv)

Video-grounded Dialogue (VGD) aims to decode an answer sentence to a question regarding a given video and dialogue context. Despite the recent success of multi-modal reasoning to generate answer sentences, existing dialogue systems still suffer from a text hallucination problem, which denotes indiscriminate text-copying from input texts without an understanding of the question. This is due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts, thus the VGD system excessively relies on copying words from input texts by hoping those words to overlap with ground-truth texts. Hence, we design Text Hallucination Mitigating (THAM) framework, which incorporates Text Hallucination Regularization (THR) loss derived from the proposed information-theoretic text hallucination measurement approach. Applying THAM with current dialogue systems validates the effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows enhanced interpretability.