mlsanityData to 7 Oct 2026

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

arXiv 2002.00163 ↗First posted 1 Feb 2020cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 7; 2023: 2.

20 papers.

Sort:

By area:

  1. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  2. 2021VidTr: Video Transformer Without Convolutionscited by 112
  3. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  4. 2022Learning to Retrieve Videos by Asking Questionscited by 17
  5. 2021Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learningcited by 15
  6. 2021DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialoguecited by 13
  7. 2021Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmarkcited by 12
  8. 2021Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialoguescited by 11
  9. 2021VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Taskscited by 11
  10. 2023VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitionscited by 11
  11. 2022Multimodal Dialogue State Trackingcited by 6
  12. 2022End-to-End Multimodal Representation Learning for Video Dialogcited by 5
8 more
  1. 2022Audio Visual Scene-Aware Dialog Generation with Transformer-based Video Representationscited by 4
  2. 2022Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generationcited by 3
  3. 2021C3: Compositional Counterfactual Contrastive Learning for Video-grounded Dialoguescited by 2
  4. 2021Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answeringcited by 1
  5. 2022Multimodal Transformer for Parallel Concatenated Variational Autoencoderscited by 1
  6. 2021WeChat AI & ICT's Submission for DSTC9 Interactive Dialogue Evaluation Trackcited by 0
  7. 2021Transfer training from smaller language modelcited by 0
  8. 2023EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performancecited by 0