mlsanityData to 14 Oct 2026

VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions

arXiv 2305.18756 ↗First posted 30 May 2023cs.CV cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Video-grounded dialogue understanding is a challenging problem that requires machine to perceive, parse and reason over situated semantics extracted from weakly aligned video and dialogues. Most existing benchmarks treat both modalities the same as a frame-independent visual understanding task, while neglecting the intrinsic attributes in multimodal dialogues, such as scene and topic transitions. In this paper, we present Video-grounded Scene&Topic AwaRe dialogue (VSTAR) dataset, a large scale video-grounded dialogue understanding dataset based on 395 TV series. Based on VSTAR, we propose two benchmarks for video-grounded dialogue understanding: scene segmentation and topic segmentation, and one benchmark for video-grounded dialogue generation. Comprehensive experiments are performed on these benchmarks to demonstrate the importance of multimodal information and segments in video-grounded dialogue understanding and generation.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

29 of 40 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2016Visual Dialogcited by 377
  10. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  11. 2016Movie Descriptioncited by 181
  12. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
17 more
  1. 2019Audio-Visual Scene-Aware Dialogcited by 123
  2. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
  3. 2020A Local-to-Global Approach to Multi-modal Movie Scene Segmentationcited by 60
  4. 2020Topic-Aware Multi-turn Dialogue Modelingcited by 46
  5. 2021Shot Contrastive Self-Supervised Learning for Scene Boundary Detectioncited by 35
  6. 2021Multimodal Dialogue Response Generationcited by 32
  7. 2021Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoringcited by 29
  8. 2017Visual Reference Resolution using Attention Memory for Visual Dialogcited by 28
  9. 2020OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contextscited by 27
  10. 2019Reasoning Visual Dialogs with Structural and Partial Observationscited by 25
  11. 2020Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialogcited by 20
  12. 2017Evaluating Visual Conversational Agents via Cooperative Human-AI Gamescited by 17
  13. 2022Boundary-aware Self-supervised Learning for Video Scene Segmentationcited by 17
  14. 2021OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contextscited by 16
  15. 2018Game-Based Video-Context Dialoguecited by 15
  16. 2021DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialoguecited by 13
  17. 2021Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 4; 2026: 1.

11 papers.

  1. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  2. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
  3. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  4. 2024Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understandingcited by 9
  5. 2026Motion-o: Trajectory-Grounded Video Reasoningcited by 4
  6. 2025Towards Fine-Grained Video Question Answeringcited by 2
  7. 2023MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learningcited by 1
  8. 2023Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialogcited by 1
  9. 2024Multimodal Fusion and Coherence Modeling for Video Topic Segmentationcited by 1
  10. 2024MTP: A Dataset for Multi-Modal Turning Points in Casual Conversationscited by 0
  11. 2025A Video-grounded Dialogue Dataset and Metric for Event-driven Activitiescited by 0