# Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog

arXiv: https://arxiv.org/abs/2310.07259 · HTML: https://mlsanity.com/paper/2310.07259/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 75 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 30 of 35)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models](https://mlsanity.com/paper/2106.00245/) (2021)
- [VD-BERT: A Unified Vision and Dialog Transformer with BERT](https://mlsanity.com/paper/2004.13278/) (2020)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [Factor Graph Attention](https://mlsanity.com/paper/1904.05880/) (2019)
- [End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features](https://mlsanity.com/paper/1806.08409/) (2018)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues](https://mlsanity.com/paper/2010.10095/) (2020)
- [Focal Visual-Text Attention for Visual Question Answering](https://mlsanity.com/paper/1806.01873/) (2018)
- [UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog](https://mlsanity.com/paper/2205.00423/) (2022)
- [Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue](https://mlsanity.com/paper/2212.05765/) (2022)
- [Structured Co-reference Graph Attention for Video-grounded Dialogue](https://mlsanity.com/paper/2103.13361/) (2021)
- [UniSA: Unified Generative Framework for Sentiment Analysis](https://mlsanity.com/paper/2309.01339/) (2023)
- [Iterative Context-Aware Graph Inference for Visual Dialog](https://mlsanity.com/paper/2004.02194/) (2020)
- [Video-Grounded Dialogues with Pretrained Generation Language Models](https://mlsanity.com/paper/2006.15319/) (2020)
- [Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System](https://mlsanity.com/paper/2001.06206/) (2020)
- [Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues](https://mlsanity.com/paper/2103.00820/) (2021)
- [VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks](https://mlsanity.com/paper/2104.07921/) (2021)
- [Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model](https://mlsanity.com/paper/2207.07934/) (2022)
- [VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions](https://mlsanity.com/paper/2305.18756/) (2023)
- [DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generator](https://mlsanity.com/paper/2004.08299/) (2020)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation](https://mlsanity.com/paper/1908.08191/) (2019)
- [Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge](https://mlsanity.com/paper/2002.10695/) (2020)

## Built on by (most cited first, 1 of 1)

- [MARS: Technical Report for the CASTLE Challenge at EgoVis 2026](https://mlsanity.com/paper/2605.18176/) (2026)

## Abstract (arXiv)

In contrast to conventional visual question answering, video-grounded dialog necessitates a profound understanding of both dialog history and video content for accurate response generation. Despite commendable progress made by existing approaches, they still face the challenges of incrementally understanding complex dialog history and assimilating video information. In response to these challenges, we present an iterative search and reasoning framework, which consists of a textual encoder, a visual encoder, and a generator. Specifically, we devise a path search and aggregation strategy in the textual encoder, mining core cues from dialog history that are pivotal to understanding the posed questions. Concurrently, our visual encoder harnesses an iterative reasoning network to extract and emphasize critical visual markers from videos, enhancing the depth of visual comprehension. Finally, we utilize the pre-trained GPT-2 model as our answer generator to decode the mined hidden clues into coherent and contextualized answers. Extensive experiments on three public datasets demonstrate the effectiveness and generalizability of our proposed framework.