# BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues

arXiv: https://arxiv.org/abs/2010.10095 · HTML: https://mlsanity.com/paper/2010.10095/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [Learning to Retrieve Videos by Asking Questions](https://mlsanity.com/paper/2205.05739/) (2022)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue](https://mlsanity.com/paper/2101.00151/) (2021)
- [VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks](https://mlsanity.com/paper/2104.07921/) (2021)
- [Video Dialog as Conversation about Objects Living in Space-Time](https://mlsanity.com/paper/2207.03656/) (2022)
- [FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory](https://mlsanity.com/paper/2308.10170/) (2023)
- [Maintaining Common Ground in Dynamic Environments](https://mlsanity.com/paper/2105.14207/) (2021)
- [Multimodal Dialogue State Tracking](https://mlsanity.com/paper/2206.07898/) (2022)
- [Audio Visual Scene-Aware Dialog Generation with Transformer-based Video Representations](https://mlsanity.com/paper/2202.09979/) (2022)
- [From Category to Scenery: An End-to-End Framework for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2407.00917/) (2024)
- [Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation](https://mlsanity.com/paper/2210.12460/) (2022)
- [M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation](https://mlsanity.com/paper/2402.11875/) (2024)
- [Multi-Modal Video Dialog State Tracking in the Wild](https://mlsanity.com/paper/2407.02218/) (2024)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [C3: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues](https://mlsanity.com/paper/2106.08914/) (2021)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog](https://mlsanity.com/paper/2310.07259/) (2023)
- [Grounding is All You Need? Dual Temporal Grounding for Video Dialog](https://mlsanity.com/paper/2410.05767/) (2024)
- [V2Dial: Unification of Video and Visual Dialog via Multimodal Experts](https://mlsanity.com/paper/2503.02063/) (2025)

## Abstract (arXiv)

Video-grounded dialogues are very challenging due to (i) the complexity of videos which contain both spatial and temporal variations, and (ii) the complexity of user utterances which query different segments and/or different objects in videos over multiple dialogue turns. However, existing approaches to video-grounded dialogues often focus on superficial temporal-level visual cues, but neglect more fine-grained spatial signals from videos. To address this drawback, we propose Bi-directional Spatio-Temporal Learning (BiST), a vision-language neural framework for high-resolution queries in videos based on textual cues. Specifically, our approach not only exploits both spatial and temporal-level information, but also learns dynamic information diffusion between the two feature spaces through spatial-to-temporal and temporal-to-spatial reasoning. The bidirectional strategy aims to tackle the evolving semantics of user queries in the dialogue setting. The retrieved visual cues are used as contextual information to construct relevant responses to the users. Our empirical results and comprehensive qualitative analysis show that BiST achieves competitive performance and generates reasonable responses on a large-scale AVSD benchmark. We also adapt our BiST models to the Video QA setting, and substantially outperform prior approaches on the TGIF-QA benchmark.