# Game-Based Video-Context Dialogue

arXiv: https://arxiv.org/abs/1809.04560 · HTML: https://mlsanity.com/paper/1809.04560/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 15 of 15)

- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset](https://mlsanity.com/paper/2212.04119/) (2022)
- [VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions](https://mlsanity.com/paper/2305.18756/) (2023)
- [Maintaining Common Ground in Dynamic Environments](https://mlsanity.com/paper/2105.14207/) (2021)
- [Going for GOAL: A Resource for Grounded Football Commentaries](https://mlsanity.com/paper/2211.04534/) (2022)
- [CAISE: Conversational Agent for Image Search and Editing](https://mlsanity.com/paper/2202.11847/) (2022)
- [Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation](https://mlsanity.com/paper/2210.12460/) (2022)
- [Commentary Generation from Data Records of Multiplayer Strategy Esports Game](https://mlsanity.com/paper/2212.10935/) (2022)
- [MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation](https://mlsanity.com/paper/2311.12820/) (2023)
- [M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation](https://mlsanity.com/paper/2402.11875/) (2024)
- [CausalDialogue: Modeling Utterance-level Causality in Conversations](https://mlsanity.com/paper/2212.10515/) (2022)
- [Semantic Frame Aggregation-based Transformer for Live Video Comment Generation](https://mlsanity.com/paper/2510.26978/) (2025)
- [A Video-grounded Dialogue Dataset and Metric for Event-driven Activities](https://mlsanity.com/paper/2501.18324/) (2025)

## Abstract (arXiv)

Current dialogue systems focus more on textual and speech context knowledge and are usually based on two speakers. Some recent work has investigated static image-based dialogue. However, several real-world human interactions also involve dynamic visual context (similar to videos) as well as dialogue exchanges among multiple speakers. To move closer towards such multimodal conversational skills and visually-situated applications, we introduce a new video-context, many-speaker dialogue dataset based on live-broadcast soccer game videos and chats from Twitch.tv. This challenging testbed allows us to develop visually-grounded dialogue models that should generate relevant temporal and spatial event language from the live video, while also being relevant to the chat history. For strong baselines, we also present several discriminative and generative models, e.g., based on tridirectional attention flow (TriDAF). We evaluate these models via retrieval ranking-recall, automatic phrase-matching metrics, as well as human evaluation studies. We also present dataset analyses, model ablations, and visualizations to understand the contribution of different modalities and model components.