# Progressive Attention Memory Network for Movie Story Question Answering

arXiv: https://arxiv.org/abs/1904.08607 · HTML: https://mlsanity.com/paper/1904.08607/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2019.

## Built on by (most cited first, 21 of 21)

- [All in One: Exploring Unified Video-Language Pre-training](https://mlsanity.com/paper/2203.07303/) (2022)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [Multi-Modal Answer Validation for Knowledge-Based VQA](https://mlsanity.com/paper/2103.12248/) (2021)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering](https://mlsanity.com/paper/2104.14085/) (2021)
- [From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering](https://mlsanity.com/paper/2205.14895/) (2022)
- [Clover: Towards A Unified Video-Language Alignment and Fusion Model](https://mlsanity.com/paper/2207.07885/) (2022)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Structured Co-reference Graph Attention for Video-grounded Dialogue](https://mlsanity.com/paper/2103.13361/) (2021)
- [Learning Video Context as Interleaved Multimodal Sequences](https://mlsanity.com/paper/2407.21757/) (2024)
- [Recent Advances in Video Question Answering: A Review of Datasets and Methods](https://mlsanity.com/paper/2101.05954/) (2021)
- [Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering](https://mlsanity.com/paper/2205.04061/) (2022)
- [On the hidden treasure of dialog in video question answering](https://mlsanity.com/paper/2103.14517/) (2021)
- [Long Story Short: a Summarize-then-Search Method for Long Video Question Answering](https://mlsanity.com/paper/2311.01233/) (2023)
- [Towards a Formal Model of Narratives](https://mlsanity.com/paper/2103.12872/) (2021)
- [NEWSKVQA: Knowledge-Aware News Video Question Answering](https://mlsanity.com/paper/2202.04015/) (2022)
- [Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling](https://mlsanity.com/paper/2210.03941/) (2022)
- [Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering](https://mlsanity.com/paper/2109.04735/) (2021)
- [Movie Genre Classification by Language Augmentation and Shot Sampling](https://mlsanity.com/paper/2203.13281/) (2022)
- [Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering](https://mlsanity.com/paper/2209.03609/) (2022)
- [Cross-Modal Reasoning with Event Correlation for Video Question Answering](https://mlsanity.com/paper/2312.12721/) (2023)

## Abstract (arXiv)

This paper proposes the progressive attention memory network (PAMN) for movie story question answering (QA). Movie story QA is challenging compared to VQA in two aspects: (1) pinpointing the temporal parts relevant to answer the question is difficult as the movies are typically longer than an hour, (2) it has both video and subtitle where different questions require different modality to infer the answer. To overcome these challenges, PAMN involves three main features: (1) progressive attention mechanism that utilizes cues from both question and answer to progressively prune out irrelevant temporal parts in memory, (2) dynamic modality fusion that adaptively determines the contribution of each modality for answering the current question, and (3) belief correction answering scheme that successively corrects the prediction score on each candidate answer. Experiments on publicly available benchmark datasets, MovieQA and TVQA, demonstrate that each feature contributes to our movie story QA architecture, PAMN, and improves performance to achieve the state-of-the-art result. Qualitative analysis by visualizing the inference mechanism of PAMN is also provided.