# Multimodal Dual Attention Memory for Video Story Question Answering

arXiv: https://arxiv.org/abs/1809.07999 · HTML: https://mlsanity.com/paper/1809.07999/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2018.

## Built on by (most cited first, 20 of 20)

- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering](https://mlsanity.com/paper/2104.14085/) (2021)
- [From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering](https://mlsanity.com/paper/2205.14895/) (2022)
- [Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering](https://mlsanity.com/paper/2106.10446/) (2021)
- [Video Question Answering with Iterative Video-Text Co-Tokenization](https://mlsanity.com/paper/2208.00934/) (2022)
- [Recent Advances in Video Question Answering: A Review of Datasets and Methods](https://mlsanity.com/paper/2101.05954/) (2021)
- [WildQA: In-the-Wild Video Question Answering](https://mlsanity.com/paper/2209.06650/) (2022)
- [Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition](https://mlsanity.com/paper/2101.03287/) (2021)
- [On the hidden treasure of dialog in video question answering](https://mlsanity.com/paper/2103.14517/) (2021)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2210.12617/) (2022)
- [LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering](https://mlsanity.com/paper/2111.14547/) (2021)
- [Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering](https://mlsanity.com/paper/2307.13250/) (2023)
- [Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answering](https://mlsanity.com/paper/2205.06530/) (2022)
- [M2FN: Multi-step Modality Fusion for Advertisement Image Assessment](https://mlsanity.com/paper/2102.00441/) (2021)
- [A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning](https://mlsanity.com/paper/2306.15977/) (2023)
- [Cross-Modal Reasoning with Event Correlation for Video Question Answering](https://mlsanity.com/paper/2312.12721/) (2023)

## Abstract (arXiv)

We propose a video story question-answering (QA) architecture, Multimodal Dual Attention Memory (MDAM). The key idea is to use a dual attention mechanism with late fusion. MDAM uses self-attention to learn the latent concepts in scene frames and captions. Given a question, MDAM uses the second attention over these latent concepts. Multimodal fusion is performed after the dual attention processes (late fusion). Using this processing pipeline, MDAM learns to infer a high-level vision-language joint representation from an abstraction of the full video content. We evaluate MDAM on PororoQA and MovieQA datasets which have large-scale QA annotations on cartoon videos and movies, respectively. For both datasets, MDAM achieves new state-of-the-art results with significant margins compared to the runner-up models. We confirm the best performance of the dual attention mechanism combined with late fusion by ablation studies. We also perform qualitative analysis by visualizing the inference mechanisms of MDAM.