# Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline

arXiv: https://arxiv.org/abs/2405.14040 · HTML: https://mlsanity.com/paper/2405.14040/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2024.

## Built on by (most cited first, 19 of 19)

- [Rolling Forcing: Autoregressive Long Video Diffusion in Real Time](https://mlsanity.com/paper/2509.25161/) (2025)
- [Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models](https://mlsanity.com/paper/2504.12626/) (2025)
- [Long Context Tuning for Video Generation](https://mlsanity.com/paper/2503.10589/) (2025)
- [Mixture of Contexts for Long Video Generation](https://mlsanity.com/paper/2508.21058/) (2025)
- [Stable Video Infinity: Infinite-Length Video Generation with Error Recycling](https://mlsanity.com/paper/2510.09212/) (2025)
- [AdsQA: Towards Advertisement Video Understanding](https://mlsanity.com/paper/2509.08621/) (2025)
- [Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding](https://mlsanity.com/paper/2604.11244/) (2026)
- [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024)
- [Text-to-Edit: Controllable End-to-End Video Ad Creation via Multimodal LLMs](https://mlsanity.com/paper/2501.05884/) (2025)
- [KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Cultures](https://mlsanity.com/paper/2410.19419/) (2024)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments](https://mlsanity.com/paper/2507.09693/) (2025)
- [VCoME: Verbal Video Composition with Multimodal Editing Effects](https://mlsanity.com/paper/2407.04697/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook](https://mlsanity.com/paper/2509.14142/) (2025)
- [HiVid-Narrator: Hierarchical Video Narrative Generation with Scene-Primed ASR-anchored Compression](https://mlsanity.com/paper/2601.07366/) (2026)
- [ActivityForensics: A Comprehensive Benchmark for Localizing Manipulated Activity in Videos](https://mlsanity.com/paper/2604.03819/) (2026)
- [Long text outline generation: Chinese text outline based on unsupervised framework and large language mode](https://mlsanity.com/paper/2412.00810/) (2024)
- [Movie2Story: A framework for understanding videos and telling stories in the form of novel text](https://mlsanity.com/paper/2412.14965/) (2024)

## Abstract (arXiv)

Video storytelling is engaging multimedia content that utilizes video and its accompanying narration to attract the audience, where a key challenge is creating narrations for recorded visual scenes. Previous studies on dense video captioning and video story generation have made some progress. However, in practical applications, we typically require synchronized narrations for ongoing visual scenes. In this work, we introduce a new task of Synchronized Video Storytelling, which aims to generate synchronous and informative narrations for videos. These narrations, associated with each video clip, should relate to the visual content, integrate relevant knowledge, and have an appropriate word count corresponding to the clip's duration. Specifically, a structured storyline is beneficial to guide the generation process, ensuring coherence and integrity. To support the exploration of this task, we introduce a new benchmark dataset E-SyncVidStory with rich annotations. Since existing Multimodal LLMs are not effective in addressing this task in one-shot or few-shot settings, we propose a framework named VideoNarrator that can generate a storyline for input videos and simultaneously generate narrations with the guidance of the generated or predefined storyline. We further introduce a set of evaluation metrics to thoroughly assess the generation. Both automatic and human evaluations validate the effectiveness of our approach. Our dataset, codes, and evaluations will be released.