mlsanityData to 14 Oct 2026

MHMS: Multimodal Hierarchical Multimedia Summarization

arXiv 2204.03734 ↗First posted 7 Apr 2022cs.CV cs.CL cs.MMcited by 6

Not on the map. Why?

Abstract (arXiv)

Multimedia summarization with multimodal output can play an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. In this work, we propose a multimodal hierarchical multimedia summarization (MHMS) framework by interacting visual and language domains to generate both video and textual summaries. Our MHMS method contains video and textual segmentation and summarization module, respectively. It formulates a cross-domain alignment objective with optimal transport distance which leverages cross-domain interaction to generate the representative keyframe and textual summary. We evaluated MHMS on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 60 of its 108 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

60 of 108 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2017Graph Attention Networkscited by 8,923
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  9. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
38 more
  1. 2019Text Summarization with Pretrained Encoderscited by 556
  2. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  3. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  4. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  5. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  6. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  7. 2017Temporal Action Detection with Structured Segment Networkscited by 267
  8. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  9. 2018Exploring Visual Relationship for Image Captioningcited by 215
  10. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  11. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  12. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
  13. 2016Neural Summarization by Extracting Sentences and Wordscited by 116
  14. 2017Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Rewardcited by 105
  15. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  16. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  17. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  18. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  19. 2021Video Summarization Using Deep Neural Networks: A Surveycited by 85
  20. 2018Text Segmentation as a Supervised Learning Taskcited by 76
  21. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  22. 2017Video Summarization with Attention-Based Encoder-Decoder Networkscited by 63
  23. 2022Multi-modal Alignment using Representation Codebookcited by 63
  24. 2016End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answeringcited by 60
  25. 2020A Local-to-Global Approach to Multi-modal Movie Scene Segmentationcited by 60
  26. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  27. 2020Text Segmentation by Cross Segment Attentioncited by 54
  28. 2018A Unified Model for Extractive and Abstractive Summarization using Inconsistency Losscited by 47
  29. 2020VMSMO: Learning to Generate Multimodal Summary for Video-based News Articlescited by 44
  30. 2016Learning Language-Visual Embedding for Movie Understanding with Natural-Languagecited by 41
  31. 2018A Perceptual Prediction Framework for Self Supervised Event Segmentationcited by 40
  32. 2019Hierarchical Optimal Transport for Multimodal Distribution Alignmentcited by 38
  33. 2018End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Featurescited by 37
  34. 2019A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentationcited by 36
  35. 2021Shot Contrastive Self-Supervised Learning for Scene Boundary Detectioncited by 35
  36. 2019Image-Question-Answer Synergistic Network for Visual Dialogcited by 23
  37. 2021Cross-domain Activity Recognition via Substructural Optimal Transportcited by 21
  38. 2018Toward Fast and Accurate Neural Discourse Segmentationcited by 19

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4.

6 papers.

  1. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  2. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  3. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  4. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  5. 2022GeoECG: Data Augmentation via Wasserstein Geodesic Perturbation for Robust Electrocardiogram Predictioncited by 8
  6. 2023Fusing Multimodal Signals on Hyper-complex Space for Extreme Abstractive Text Summarization (TL;DR) of Scientific Contentscited by 7