# M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation

arXiv: https://arxiv.org/abs/2402.11875 · HTML: https://mlsanity.com/paper/2402.11875/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [Temporal Reasoning via Audio Question Answering](https://mlsanity.com/paper/1911.09655/) (2019)
- [BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues](https://mlsanity.com/paper/2010.10095/) (2020)
- [Game-Based Video-Context Dialogue](https://mlsanity.com/paper/1809.04560/) (2018)
- [Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning](https://mlsanity.com/paper/2110.06894/) (2021)
- [HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue](https://mlsanity.com/paper/2312.09736/) (2023)
- [Audio Visual Scene-Aware Dialog Generation with Transformer-based Video Representations](https://mlsanity.com/paper/2202.09979/) (2022)
- [Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation](https://mlsanity.com/paper/2210.12460/) (2022)
- [MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation](https://mlsanity.com/paper/2311.12820/) (2023)
- [C3: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues](https://mlsanity.com/paper/2106.08914/) (2021)

## Built on by (most cited first, 3 of 3)

- [Question-Aware Gaussian Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2503.04459/) (2025)
- [Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs](https://mlsanity.com/paper/2604.05522/) (2026)
- [Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2511.23304/) (2025)

## Abstract (arXiv)

Video-grounded dialogue generation (VDG) requires the system to generate a fluent and accurate answer based on multimodal knowledge. However, the difficulty in multimodal knowledge utilization brings serious hallucinations to VDG models in practice. Although previous works mitigate the hallucination in a variety of ways, they hardly take notice of the importance of the multimodal knowledge anchor answer tokens. In this paper, we reveal via perplexity that different VDG models experience varying hallucinations and exhibit diverse anchor tokens. Based on this observation, we propose M2K-VDG, a model-adaptive multimodal knowledge anchor enhancement framework for hallucination reduction. Furthermore, we introduce the counterfactual effect for more accurate anchor token detection. The experimental results on three popular benchmarks exhibit the superiority of our approach over state-of-the-art methods, demonstrating its effectiveness in reducing hallucinations.