mlsanityData to 14 Oct 2026

Video Dialog as Conversation about Objects Living in Space-Time

arXiv 2207.03656 ↗First posted 8 Jul 2022cs.CV cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

It would be a technological feat to be able to create a system that can hold a meaningful conversation with humans about what they watch. A setup toward that goal is presented as a video dialog task, where the system is asked to generate natural utterances in response to a question in an ongoing dialog. The task poses great visual, linguistic, and reasoning challenges that cannot be easily overcome without an appropriate representation scheme over video and dialog that supports high-level reasoning. To tackle these challenges we present a new object-centric framework for video dialog that supports neural reasoning dubbed COST - which stands for Conversation about Objects in Space-Time. Here dynamic space-time visual content in videos is first parsed into object trajectories. Given this video abstraction, COST maintains and tracks object-associated dialog states, which are updated upon receiving new questions. Object interactions are dynamically and conditionally inferred for each question, and these serve as the basis for relational reasoning among them. COST also maintains a history of previous answers, and this allows retrieval of relevant object-centric information to enrich the answer forming process. Language production then proceeds in a step-wise manner, taking into the context of the current utterance, the existing dialog, the current question. We evaluate COST on the DSTC7 and DSTC8 benchmarks, demonstrating its competitiveness against state-of-the-arts.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 8 Jul 2022.

Builds on

47 of 55 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  9. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 910
  10. 2015Pointer Networkscited by 895
  11. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  12. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
35 more
  1. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  2. 2016Visual Dialogcited by 377
  3. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  4. 2018Videos as Space-Time Region Graphscited by 221
  5. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  6. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  7. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  8. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  9. 2019Audio-Visual Scene-Aware Dialogcited by 123
  10. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  11. 2020Location-aware Graph Convolutional Networks for Video Question Answeringcited by 80
  12. 2017Action Tubelet Detector for Spatio-Temporal Action Localizationcited by 77
  13. 2018Object Level Visual Reasoning in Videoscited by 67
  14. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  15. 2019Dialog State Tracking: A Neural Reading Comprehension Approachcited by 62
  16. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 61
  17. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 55
  18. 2018Visual Coreference Resolution in Visual Dialog using Neural Module Networkscited by 54
  19. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  20. 2018End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Featurescited by 37
  21. 2021Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answeringcited by 29
  22. 2017Visual Reference Resolution using Attention Memory for Visual Dialogcited by 28
  23. 2019The Eighth Dialog System Technology Challengecited by 25
  24. 2020BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialoguescited by 22
  25. 2021Structured Co-reference Graph Attention for Video-grounded Dialoguecited by 17
  26. 2018From FiLM to Video: Multi-turn Question Answering with Multi-modal Contextcited by 12
  27. 2019Dialog System Technology Challenge 7cited by 11
  28. 2020Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue Systemcited by 11
  29. 2021Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialoguescited by 11
  30. 2020DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generatorcited by 9
  31. 2018Object-based reasoning in VQAcited by 7
  32. 2019Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generationcited by 7
  33. 2020Dynamic Language Binding in Relational Visual Reasoningcited by 7
  34. 2020Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challengecited by 5
  35. 2019Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modelingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 2; 2026: 2.

11 papers.

  1. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  2. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  3. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  4. 2023HEAR: Hearing Enhanced Audio Response for Video-grounded Dialoguecited by 13
  5. 2023FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memorycited by 11
  6. 2025PAVE: Patching and Adapting Video Large Language Modelscited by 4
  7. 2024Multi-Modal Video Dialog State Tracking in the Wildcited by 3
  8. 2024Grounding is All You Need? Dual Temporal Grounding for Video Dialogcited by 1
  9. 2025V2Dial: Unification of Video and Visual Dialog via Multimodal Expertscited by 0
  10. 2026V-LynX: Token Interface Alignment for Video+X LLMscited by 0
  11. 2026Companion-style QA Assistance in Ego-Visioncited by 0