# Video Storytelling: Textual Summaries for Events

arXiv: https://arxiv.org/abs/1807.09418 · HTML: https://mlsanity.com/paper/1807.09418/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2018.

## Built on by (most cited first, 22 of 22)

- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [LMEye: An Interactive Perception Network for Large Language Models](https://mlsanity.com/paper/2305.03701/) (2023)
- [Image Retrieval from Contextual Descriptions](https://mlsanity.com/paper/2203.15867/) (2022)
- [What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation](https://mlsanity.com/paper/2408.14622/) (2024)
- [A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot](https://mlsanity.com/paper/2305.09758/) (2023)
- [Progress-Aware Video Frame Captioning](https://mlsanity.com/paper/2412.02071/) (2024)
- [What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations](https://mlsanity.com/paper/2502.08279/) (2025)
- [ViCo: Engaging Video Comment Generation with Human Preference Rewards](https://mlsanity.com/paper/2308.11171/) (2023)
- [Album Storytelling with Iterative Story-aware Captioning and Large Language Models](https://mlsanity.com/paper/2305.12943/) (2023)
- [Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning](https://mlsanity.com/paper/2507.20163/) (2025)
- [Communication breakdown: On the low mutual intelligibility between human and neural captioning](https://mlsanity.com/paper/2210.11512/) (2022)
- [A Neural Divide-and-Conquer Reasoning Framework for Image Retrieval from Linguistically Complex Text](https://mlsanity.com/paper/2305.02265/) (2023)
- [Pragmatic Inference with a CLIP Listener for Contrastive Captioning](https://mlsanity.com/paper/2306.08818/) (2023)
- [Multi-Modal Experience Inspired AI Creation](https://mlsanity.com/paper/2209.02427/) (2022)
- [Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering](https://mlsanity.com/paper/2307.13250/) (2023)
- [The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective](https://mlsanity.com/paper/2405.08720/) (2024)
- [Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark](https://mlsanity.com/paper/2401.13888/) (2024)
- [Fine-grained length controllable video captioning with ordinal embeddings](https://mlsanity.com/paper/2408.15447/) (2024)
- [A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video](https://mlsanity.com/paper/2312.01575/) (2023)
- [HiVid-Narrator: Hierarchical Video Narrative Generation with Scene-Primed ASR-anchored Compression](https://mlsanity.com/paper/2601.07366/) (2026)
- [Movie2Story: A framework for understanding videos and telling stories in the form of novel text](https://mlsanity.com/paper/2412.14965/) (2024)
- [Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model](https://mlsanity.com/paper/2606.17950/) (2026)

## Abstract (arXiv)

Bridging vision and natural language is a longstanding goal in computer vision and multimedia research. While earlier works focus on generating a single-sentence description for visual content, recent works have studied paragraph generation. In this work, we introduce the problem of video storytelling, which aims at generating coherent and succinct stories for long videos. Video storytelling introduces new challenges, mainly due to the diversity of the story and the length and complexity of the video. We propose novel methods to address the challenges. First, we propose a context-aware framework for multimodal embedding learning, where we design a Residual Bidirectional Recurrent Neural Network to leverage contextual information from past and future. Second, we propose a Narrator model to discover the underlying storyline. The Narrator is formulated as a reinforcement learning agent which is trained by directly optimizing the textual metric of the generated story. We evaluate our method on the Video Story dataset, a new dataset that we have collected to enable the study. We compare our method with multiple state-of-the-art baselines, and show that our method achieves better performance, in terms of quantitative measures and user study.