mlsanityData to 14 Oct 2026

Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding

arXiv 2203.05711 ↗First posted 11 Mar 2022cs.CV cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Despite recent advances of AI, story understanding remains an open and under-investigated problem. We collect, preprocess, and publicly release a video-language story dataset, Synopses of Movie Narratives (SyMoN), containing 5,193 video summaries of popular movies and TV series with a total length of 869 hours. SyMoN captures naturalistic storytelling videos made by human creators and intended for a human audience. As a prototypical and naturalistic story dataset, SyMoN features high coverage of multimodal story events and abundant mental-state descriptions. Its use of storytelling techniques cause cross-domain semantic gaps that provide appropriate challenges to existing models. We establish benchmarks on video-text retrieval and zero-shot alignment on movie summary videos, which showcase the importance of in-domain data and long-term memory in story understanding. With SyMoN, we hope to lay the groundwork for progress in multimodal story understanding.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 61 of its 141 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

61 of 141 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2015Fast R-CNNcited by 2,933
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2017The Kinetics Human Action Video Datasetcited by 1,909
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  11. 2017Dense-Captioning Events in Videoscited by 793
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
38 more
  1. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  3. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  4. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  5. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  8. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  9. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  10. 2019Abductive Commonsense Reasoningcited by 301
  11. 2019Character Region Awareness for Text Detectioncited by 278
  12. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  13. 2015A Dataset for Movie Descriptioncited by 195
  14. 2016Movie Descriptioncited by 181
  15. 2016Visual Storytellingcited by 176
  16. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
  17. 2021Towards Long-Form Video Understandingcited by 146
  18. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  19. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 108
  20. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  21. 2018Event2Mind: Commonsense Inference on Events, Intents, and Reactionscited by 82
  22. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 79
  23. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 68
  24. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  25. 2020TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questionscited by 65
  26. 2017Event Representations for Automated Story Generation with Deep Neural Netscited by 63
  27. 2014Narrative Planning: Balancing Plot and Charactercited by 61
  28. 2018Modeling Naive Psychology of Characters in Simple Commonsense Storiescited by 60
  29. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  30. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  31. 2020Joint Constrained Learning for Event-Event Relation Extractioncited by 53
  32. 2021"Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understandingcited by 48
  33. 2016The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narrativescited by 42
  34. 2015Using Descriptive Video Services to Create a Large Data Source for Video Annotation Researchcited by 33
  35. 2019Frowning Frodo, Wincing Leia, and a Seriously Great Friendship: Learning to Classify Emotional Relationships of Fictional Characterscited by 23
  36. 2020Screenplay Summarization Using Latent Narrative Structurecited by 23
  37. 2020Movie Summarization via Sparse Graph Constructioncited by 22
  38. 2019Fine-Grained Temporal Relation Extractioncited by 21

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2023AutoAD: Movie Description in Contextcited by 50
  2. 2023Event Causality Is Key to Computational Story Understandingcited by 11
  3. 2024Movie101v2: Improved Movie Narration Benchmarkcited by 9
  4. 2022TeViS:Translating Text Synopses to Video Storyboardscited by 5
  5. 2023PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramascited by 4
  6. 2024Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understandingcited by 2
  7. 2025Two Causally Related Needles in a Video Haystackcited by 1
  8. 2026REFRAMED: Towards Realistic Audio Description Generation for Moviescited by 1