mlsanityData to 14 Oct 2026

A Modular Approach for Multimodal Summarization of TV Shows

arXiv 2403.03823 ↗First posted 6 Mar 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper we address the task of summarizing television shows, which touches key areas in AI research: complex reasoning, multiple modalities, and long narratives. We present a modular approach where separate components perform specialized sub-tasks which we argue affords greater flexibility compared to end-to-end methods. Our modules involve detecting scene boundaries, reordering scenes so as to minimize the number of cuts between different events, converting visual information to text, summarizing the dialogue in each scene, and fusing the scene summaries into a final summary for the entire episode. We also present a new metric, PRISMA (Precision and Recall EvaluatIon of Summary FActs), to measure both precision and recall of generated summaries, which we decompose into atomic facts. Tested on the recently released SummScreen3D dataset, our method produces higher quality summaries than comparison models, as measured with ROUGE and our new fact-based metric, and as assessed by human evaluators.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

42 of 61 references.

References · 42

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2015Teaching Machines to Read and Comprehendcited by 1,049
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  12. 2017Learning multiple visual domains with residual adapterscited by 557
30 more
  1. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  2. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  3. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  4. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  5. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  6. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  7. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  8. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  9. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  10. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  11. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  12. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  13. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 76
  14. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  15. 2021Open-book Video Captioning with Retrieve-Copy-Generate Networkcited by 43
  16. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  17. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  18. 2020VIOLIN: A Large-Scale Dataset for Video-and-Language Inferencecited by 41
  19. 2021Hierarchical Modular Network for Video Captioningcited by 32
  20. 2020Movie Summarization via Sparse Graph Constructioncited by 22
  21. 2019Analyzing Sentence Fusion in Abstractive Summarizationcited by 19
  22. 2018Structure-Infused Copy Mechanisms for Abstractive Summarizationcited by 17
  23. 2022Hierarchical3D Adapters for Long Video-to-text Summarizationcited by 12
  24. 2023A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shotcited by 11
  25. 2023Minimum Description Length Clustering to Measure Meaningful Image Complexitycited by 10
  26. 2021Selective Pseudo-label Clusteringcited by 8
  27. 2023Hard Regularization to Prevent Deep Online Clustering Collapse without Data Augmentationcited by 7
  28. 2022Efficient Deep Clustering of Human Activities and How to Improve Evaluationcited by 5
  29. 2022CREATIVESUMM: Shared Task on Automatic Summarization for Creative Writingcited by 5
  30. 2023Unsupervised Embedding Learning for Human Activity Recognition Using Wearable Sensor Datacited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  2. 2025Context-Aware Hierarchical Merging for Long Document Summarizationcited by 9
  3. 2024ScreenWriter: Automatic Screenplay Generation and Movie Summarisationcited by 5
  4. 2024BookWorm: A Dataset for Character Description and Analysiscited by 3
  5. 2025Parameter-free Video Segmentation for Vision and Language Understandingcited by 3
  6. 2025K*-Means: A Parameter-free Clustering Algorithmcited by 3
  7. 2024Towards a Universal Method for Meaningful Signal Detectioncited by 2
  8. 2025Local Compositional Complexity: How to Detect a Human-readable Messsagecited by 2
  9. 2025Enhancing Long Document Long Form Summarisation with Self-Planningcited by 1
  10. 2026S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literaturecited by 1
  11. 2025Integrating Video and Text: A Balanced Approach to Multimodal Summary Generation and Evaluationcited by 0
  12. 2025Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extractioncited by 0
1 more
  1. 2026MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Awarecited by 0