# Multimodal Dialogue State Tracking By QA Approach with Data Augmentation

arXiv: https://arxiv.org/abs/2007.09903 · HTML: https://mlsanity.com/paper/2007.09903/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2020.

## Built on by (most cited first, 7 of 7)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [Slot Self-Attentive Dialogue State Tracking](https://mlsanity.com/paper/2101.09374/) (2021)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Multimodal Dialogue State Tracking](https://mlsanity.com/paper/2206.07898/) (2022)
- [Multi-Modal Video Dialog State Tracking in the Wild](https://mlsanity.com/paper/2407.02218/) (2024)
- [Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning](https://mlsanity.com/paper/2407.12498/) (2024)

## Abstract (arXiv)

Recently, a more challenging state tracking task, Audio-Video Scene-Aware Dialogue (AVSD), is catching an increasing amount of attention among researchers. Different from purely text-based dialogue state tracking, the dialogue in AVSD contains a sequence of question-answer pairs about a video and the final answer to the given question requires additional understanding of the video. This paper interprets the AVSD task from an open-domain Question Answering (QA) point of view and proposes a multimodal open-domain QA system to deal with the problem. The proposed QA system uses common encoder-decoder framework with multimodal fusion and attention. Teacher forcing is applied to train a natural language generator. We also propose a new data augmentation approach specifically under QA assumption. Our experiments show that our model and techniques bring significant improvements over the baseline model on the DSTC7-AVSD dataset and demonstrate the potentials of our data augmentation techniques.