mlsanityData to 7 Oct 2026

A Dataset for Movie Description

arXiv 1501.02530 ↗First posted 12 Jan 2015cs.CV cs.CL cs.IRcited by 195

Not on the map. Why?

Abstract (arXiv)

Descriptive video service (DVS) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed DVS, which is temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the Movie Description dataset contains a parallel corpus of over 54,000 sentences and video snippets from 72 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing DVS to scripts, we find that DVS is far more visual and describes precisely what is shown rather than what should happen according to the scripts created prior to movie production.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jan 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 39; 2023: 36; 2024: 42; 2025: 29; 2026: 13.

195 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  3. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  4. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  5. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  8. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  9. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  10. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  11. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  12. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
38 more
  1. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  2. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  3. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  6. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  7. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  8. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  9. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  10. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  11. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  12. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  13. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  14. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  15. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  16. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  17. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  18. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  19. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  20. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  21. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  22. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  23. 2024VidGen-1M: A Large-Scale Dataset for Text-to-video Generationcited by 60
  24. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  25. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  26. 2023SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentariescited by 57
  27. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  28. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  29. 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
  30. 2023AutoAD: Movie Description in Contextcited by 50
  31. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  32. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  33. 2021On Semantic Similarity in Video Retrievalcited by 45
  34. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  35. 2023Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challengescited by 42
  36. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  37. 2024VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Modelscited by 40
  38. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38

50 most cited of 195 All 195 (JSON)