# Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge

arXiv: https://arxiv.org/abs/2002.10695 · HTML: https://mlsanity.com/paper/2002.10695/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2020.

## Built on by (most cited first, 5 of 5)

- [Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue](https://mlsanity.com/paper/2212.05765/) (2022)
- [Structured Co-reference Graph Attention for Video-grounded Dialogue](https://mlsanity.com/paper/2103.13361/) (2021)
- [Video Dialog as Conversation about Objects Living in Space-Time](https://mlsanity.com/paper/2207.03656/) (2022)
- [MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation](https://mlsanity.com/paper/2311.12820/) (2023)
- [Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog](https://mlsanity.com/paper/2310.07259/) (2023)

## Abstract (arXiv)

Audio-Visual Scene-Aware Dialog (AVSD) is an extension from Video Question Answering (QA) whereby the dialogue agent is required to generate natural language responses to address user queries and carry on conversations. This is a challenging task as it consists of video features of multiple modalities, including text, visual, and audio features. The agent also needs to learn semantic dependencies among user utterances and system responses to make coherent conversations with humans. In this work, we describe our submission to the AVSD track of the 8th Dialogue System Technology Challenge. We adopt dot-product attention to combine text and non-text features of input video. We further enhance the generation capability of the dialogue agent by adopting pointer networks to point to tokens from multiple source sequences in each generation step. Our systems achieve high performance in automatic metrics and obtain 5th and 6th place in human evaluation among all submissions.