mlsanityData to 7 Oct 2026

"Previously on ..." From Recaps to Story Summarization

arXiv 2405.11487 ↗First posted 19 May 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce multimodal story summarization by leveraging TV episode recaps - short video sequences interweaving key story moments from previous episodes to bring viewers up to speed. We propose PlotSnap, a dataset featuring two crime thriller TV shows with rich recaps and long episodes of 40 minutes. Story summarization labels are unlocked by matching recap shots to corresponding sub-stories in the episode. We propose a hierarchical model TaleSumm that processes entire episodes by creating compact shot and dialog representations, and predicts importance scores for each video shot and dialog utterance by enabling interactions between local story groups. Unlike traditional summarization, our method extracts multiple plot points from long videos. We present a thorough evaluation on story summarization, including promising cross-series generalization. TaleSumm also shows good results on classic video summarization benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 72 of its 122 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

72 of 122 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2016Layer Normalizationcited by 5,579
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  13. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  14. 2019Text Summarization with Pretrained Encoderscited by 553
  15. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  16. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  17. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  18. 2015A Dataset for Movie Descriptioncited by 195
  19. 2020Extractive Summarization as Text Matchingcited by 189
  20. 2015Visual Semantic Role Labelingcited by 188
  21. 2016Movie Descriptioncited by 180
  22. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  23. 2016Video Summarization with Long Short-term Memorycited by 142
  24. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  25. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  26. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  27. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  28. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  29. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  30. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  31. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  32. 2019Rethinking the Evaluation of Video Summariescited by 62
  33. 2018Learnable PINs: Cross-Modal Embeddings for Person Identitycited by 60
  34. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  35. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  36. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  37. 2014Coherent Multi-Sentence Video Description with Variable Level of Detailcited by 45
  38. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editingcited by 1