# Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research

arXiv: https://arxiv.org/abs/1503.01070 · HTML: https://mlsanity.com/paper/1503.01070/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2015.

## Built on by (most cited first, 30 of 33)

- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning](https://mlsanity.com/paper/2311.17435/) (2023)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training](https://mlsanity.com/paper/2104.09411/) (2021)
- [DistinctAD: Distinctive Audio Description Generation in Contexts](https://mlsanity.com/paper/2411.18180/) (2024)
- [Movie101v2: Improved Movie Narration Benchmark](https://mlsanity.com/paper/2404.13370/) (2024)
- [Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding](https://mlsanity.com/paper/2203.05711/) (2022)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning](https://mlsanity.com/paper/2104.00285/) (2021)
- [Distribution Aware Metrics for Conditional Natural Language Generation](https://mlsanity.com/paper/2209.07518/) (2022)
- [Detecting and Grounding Important Characters in Visual Stories](https://mlsanity.com/paper/2303.17647/) (2023)
- [OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video](https://mlsanity.com/paper/2604.11102/) (2026)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [TeViS:Translating Text Synopses to Video Storyboards](https://mlsanity.com/paper/2301.00135/) (2022)
- [Taking an Emotional Look at Video Paragraph Captioning](https://mlsanity.com/paper/2203.06356/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Beyond Grounding: Extracting Fine-Grained Event Hierarchies Across Modalities](https://mlsanity.com/paper/2206.07207/) (2022)
- [NowYouSee Me: Context-Aware Automatic Audio Description](https://mlsanity.com/paper/2412.10002/) (2024)
- [Can Vision Language Models Understand Mimed Actions?](https://mlsanity.com/paper/2506.21586/) (2025)
- [Narration Generation for Cartoon Videos](https://mlsanity.com/paper/2101.06803/) (2021)
- [The Role of the Input in Natural Language Video Description](https://mlsanity.com/paper/2102.05067/) (2021)
- [Self-Contained Entity Discovery from Captioned Videos](https://mlsanity.com/paper/2208.06662/) (2022)
- [An Integrated Approach for Video Captioning and Applications](https://mlsanity.com/paper/2201.09153/) (2022)

## Abstract (arXiv)

In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.