# Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues

arXiv: https://arxiv.org/abs/2103.00820 · HTML: https://mlsanity.com/paper/2103.00820/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 78 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions](https://mlsanity.com/paper/1602.01103/) (2016)
- [Argumentation Mining in User-Generated Web Discourse](https://mlsanity.com/paper/1601.02403/) (2016)
- [Cognitive Graph for Multi-Hop Reading Comprehension at Scale](https://mlsanity.com/paper/1905.05460/) (2019)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [Dynamically Fused Graph Network for Multi-hop Reasoning](https://mlsanity.com/paper/1905.06933/) (2019)
- [Question Answering by Reasoning Across Documents with Graph Convolutional Networks](https://mlsanity.com/paper/1808.09920/) (2018)
- [Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs](https://mlsanity.com/paper/1905.07374/) (2019)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [Factor Graph Attention](https://mlsanity.com/paper/1904.05880/) (2019)
- [AMPERSAND: Argument Mining for PERSuAsive oNline Discussions](https://mlsanity.com/paper/2004.14677/) (2020)
- [A Deep Sequential Model for Discourse Parsing on Multi-Party Dialogues](https://mlsanity.com/paper/1812.00176/) (2018)
- [Argument Mining with Structured SVMs and RNNs](https://mlsanity.com/paper/1704.06869/) (2017)

## Built on by (most cited first, 11 of 11)

- [LAVIS: A Library for Language-Vision Intelligence](https://mlsanity.com/paper/2209.09019/) (2022)
- [HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue](https://mlsanity.com/paper/2312.09736/) (2023)
- [Video Dialog as Conversation about Objects Living in Space-Time](https://mlsanity.com/paper/2207.03656/) (2022)
- [Multimodal Dialogue State Tracking](https://mlsanity.com/paper/2206.07898/) (2022)
- [Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation](https://mlsanity.com/paper/2210.12460/) (2022)
- [OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog](https://mlsanity.com/paper/2402.13146/) (2024)
- [Multi-Modal Video Dialog State Tracking in the Wild](https://mlsanity.com/paper/2407.02218/) (2024)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog](https://mlsanity.com/paper/2310.07259/) (2023)
- [Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidance](https://mlsanity.com/paper/2509.05669/) (2025)
- [V2Dial: Unification of Video and Visual Dialog via Multimodal Experts](https://mlsanity.com/paper/2503.02063/) (2025)

## Abstract (arXiv)

Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi-turn setting. Previous approaches to video-grounded dialogues mostly use dialogue context as a simple text input without modelling the inherent information flows at the turn level. In this paper, we propose a novel framework of Reasoning Paths in Dialogue Context (PDC). PDC model discovers information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. PDC model then learns to predict reasoning paths over this semantic graph. Our path prediction model predicts a path from the current turn through past dialogue turns that contain additional visual cues to answer the current question. Our reasoning model sequentially processes both visual and textual information through this reasoning path and the propagated features are used to generate the answer. Our experimental results demonstrate the effectiveness of our method and provide additional insights on how models use semantic dependencies in a dialogue context to retrieve visual cues.