mlsanityData to 7 Oct 2026

Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research

arXiv 1503.01070 ↗First posted 3 Mar 2015cs.CV cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 7; 2024: 5; 2025: 2; 2026: 2.

33 papers.

Sort:

By area:

  1. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  3. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  4. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  5. 2023AutoAD: Movie Description in Contextcited by 50
  6. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  7. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  8. 2023A Review of Deep Learning for Video Captioningcited by 28
  9. 2021Recent Advances and Trends in Multimodal Deep Learning: A Reviewcited by 26
  10. 2023Movie101: A New Movie Understanding Benchmarkcited by 25
  11. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  12. 2024DistinctAD: Distinctive Audio Description Generation in Contextscited by 11
21 more
  1. 2024Movie101v2: Improved Movie Narration Benchmarkcited by 9
  2. 2022Synopses of Movie Narratives: a Video-Language Dataset for Story Understandingcited by 8
  3. 2024ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognitioncited by 8
  4. 2021CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learningcited by 6
  5. 2022Distribution Aware Metrics for Conditional Natural Language Generationcited by 6
  6. 2023Detecting and Grounding Important Characters in Visual Storiescited by 6
  7. 2021A Comprehensive Review of the Video-to-Text Problemcited by 5
  8. 2022TeViS:Translating Text Synopses to Video Storyboardscited by 5
  9. 2026OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Videocited by 5
  10. 2022Taking an Emotional Look at Video Paragraph Captioningcited by 3
  11. 2022Video Captioning: a comparative review of where we are and which could be the routecited by 3
  12. 2022Beyond Grounding: Extracting Fine-Grained Event Hierarchies Across Modalitiescited by 3
  13. 2024NowYouSee Me: Context-Aware Automatic Audio Descriptioncited by 3
  14. 2025Can Vision Language Models Understand Mimed Actions?cited by 2
  15. 2021Narration Generation for Cartoon Videoscited by 1
  16. 2021The Role of the Input in Natural Language Video Descriptioncited by 1
  17. 2022Self-Contained Entity Discovery from Captioned Videoscited by 1
  18. 2022An Integrated Approach for Video Captioning and Applicationscited by 0
  19. 2023Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creationcited by 0
  20. 2025Data Transformation Strategies to Remove Heterogeneitycited by 0
  21. 2026StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videoscited by 0