mlsanityData to 7 Oct 2026

Audio-Visual Scene-Aware Dialog

arXiv 1901.09107 ↗First posted 25 Jan 2019cs.CVcited by 122

Not on the map. Why?

Abstract (arXiv)

We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history of previous turns in the dialog. To answer successfully, agents must ground concepts from the question in the video while leveraging contextual cues from the dialog history. To benchmark this task, we introduce the Audio Visual Scene-Aware Dialog (AVSD) Dataset. For each of more than 11,000 videos of human actions from the Charades dataset, our dataset contains a dialog about the video, plus a final summary of the video by one of the dialog participants. We train several baseline systems for this task and evaluate the performance of the trained models using both qualitative and quantitative metrics. Our results indicate that models must utilize all the available inputs (video, audio, question, and dialog history) to perform best on this dataset.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 28; 2023: 21; 2024: 18; 2025: 23; 2026: 9.

122 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  4. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  5. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  6. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  7. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  8. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  9. 2023Audio-Visual LLM for Video Understandingcited by 76
  10. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  11. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  12. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
38 more
  1. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  2. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  3. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  4. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  5. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  6. 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
  7. 2022DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generationcited by 35
  8. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  9. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  10. 2021ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generationcited by 31
  11. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  12. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
  13. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  14. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  15. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  16. 2021M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretrainingcited by 26
  17. 2023A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Modelscited by 23
  18. 2022Learning to Transfer Prompts for Text Generationcited by 22
  19. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  20. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  21. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  22. 2024Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluationscited by 19
  23. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  24. 2022Information-Theoretic Text Hallucination Reduction for Video-grounded Dialoguecited by 18
  25. 2023Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parsercited by 18
  26. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  27. 2021Structured Co-reference Graph Attention for Video-grounded Dialoguecited by 17
  28. 2022Learning to Retrieve Videos by Asking Questionscited by 17
  29. 2023Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialoguescited by 17
  30. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 16
  31. 2021OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contextscited by 16
  32. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  33. 2022Video Question Answering with Iterative Video-Text Co-Tokenizationcited by 16
  34. 2021Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learningcited by 15
  35. 2021DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialoguecited by 13
  36. 2021Video Question Answering with Phrases via Semantic Rolescited by 13
  37. 2023HEAR: Hearing Enhanced Audio Response for Video-grounded Dialoguecited by 13
  38. 2021Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmarkcited by 12

50 most cited of 122 All 122 (JSON)