# Long Story Short: a Summarize-then-Search Method for Long Video Question Answering

arXiv: https://arxiv.org/abs/2311.01233 · HTML: https://mlsanity.com/paper/2311.01233/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)
- [VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling](https://mlsanity.com/paper/2111.12681/) (2021)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Motion-Appearance Co-Memory Networks for Video Question Answering](https://mlsanity.com/paper/1803.10906/) (2018)
- [DeepStory: Video Story QA by Deep Embedded Memory Networks](https://mlsanity.com/paper/1707.00836/) (2017)
- [Leveraging Video Descriptions to Learn Video Question Answering](https://mlsanity.com/paper/1611.04021/) (2016)
- [DramaQA: Character-Centered Video Story Understanding with Hierarchical QA](https://mlsanity.com/paper/2005.03356/) (2020)
- [Movie Plot Analysis via Turning Point Identification](https://mlsanity.com/paper/1908.10328/) (2019)
- [Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization](https://mlsanity.com/paper/2208.09770/) (2022)
- [Self-supervised pre-training and contrastive representation learning for multiple-choice video QA](https://mlsanity.com/paper/2009.08043/) (2020)
- [Screenplay Summarization Using Latent Narrative Structure](https://mlsanity.com/paper/2004.12727/) (2020)
- [Progressive Attention Memory Network for Movie Story Question Answering](https://mlsanity.com/paper/1904.08607/) (2019)
- [A Read-Write Memory Network for Movie Story Understanding](https://mlsanity.com/paper/1709.09345/) (2017)
- [Are we asking the right questions in MovieQA?](https://mlsanity.com/paper/1911.03083/) (2019)

## Built on by (most cited first, 7 of 7)

- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [DrVideo: Document Retrieval Based Long Video Understanding](https://mlsanity.com/paper/2406.12846/) (2024)
- [VCA: Video Curious Agent for Long Video Understanding](https://mlsanity.com/paper/2412.10471/) (2024)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [Investigating Video Reasoning Capability of Large Language Models with Tropes in Movies](https://mlsanity.com/paper/2406.10923/) (2024)
- [FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMs](https://mlsanity.com/paper/2503.19850/) (2025)

## Abstract (arXiv)

Large language models such as GPT-3 have demonstrated an impressive capability to adapt to new tasks without requiring task-specific training data. This capability has been particularly effective in settings such as narrative question answering, where the diversity of tasks is immense, but the available supervision data is small. In this work, we investigate if such language models can extend their zero-shot reasoning abilities to long multimodal narratives in multimedia content such as drama, movies, and animation, where the story plays an essential role. We propose Long Story Short, a framework for narrative video QA that first summarizes the narrative of the video to a short plot and then searches parts of the video relevant to the question. We also propose to enhance visual matching with CLIPCheck. Our model outperforms state-of-the-art supervised models by a large margin, highlighting the potential of zero-shot QA for long videos.