mlsanityData to 7 Oct 2026

A Survey on Multi-modal Summarization

arXiv 2109.05199 ↗First posted 11 Sep 2021cs.CL cs.MM cs.NEcited by 26

Not on the map. Why?

Abstract (arXiv)

The new era of technology has brought us to the point where it is convenient for people to share their opinions over an abundance of platforms. These platforms have a provision for the users to express themselves in multiple forms of representations, including text, images, videos, and audio. This, however, makes it difficult for users to obtain all the key information about a topic, making the task of automatic multi-modal summarization (MMS) essential. In this paper, we present a comprehensive survey of the existing research in the area of MMS, covering various modalities like text, image, audio, and video. Apart from highlighting the different evaluation metrics and datasets used for the MMS task, our work also discusses the current challenges and future directions in this field.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 62 of its 202 references that are in the map. First posted on arXiv 11 Sep 2021.

Builds on

62 of 202 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  7. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  8. 2018Image Transformercited by 770
  9. 2021Going deeper with Image Transformerscited by 759
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  13. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  16. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  17. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  18. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  19. 2020Support-set bottlenecks for video-text representation learningcited by 170
  20. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  21. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  22. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  23. 2017Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Rewardcited by 105
  24. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  25. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  26. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  27. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  28. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  29. 2020Multi-document Summarization via Deep Learning Techniques: A Surveycited by 58
  30. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  31. 2020VMSMO: Learning to Generate Multimodal Summary for Video-based News Articlescited by 44
  32. 2018Multimodal Named Entity Recognition for Short Social Media Postscited by 43
  33. 2016Multimodal Residual Learning for Visual QAcited by 42
  34. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  35. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  36. 2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
  37. 2019Towards Knowledge-Based Personalized Product Description Generation in E-commercecited by 32
  38. 2018Learn to Combine Modalities in Multimodal Deep Learningcited by 29

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 2; 2025: 8; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2023Long Text and Multi-Table Summarization: Dataset and Methodcited by 23
  4. 2024Abstractive Text Summarization: State of the Art, Challenges, and Improvementscited by 17
  5. 2022A Survey on Medical Document Summarizationcited by 14
  6. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  7. 2023Large Scale Multi-Lingual Multi-Modal Summarization Datasetcited by 12
  8. 2023Transformer-based Image Generation from Scene Graphscited by 8
  9. 2023Learning to Imagine: Visually-Augmented Natural Language Generationcited by 7
  10. 2023The Current State of Summarizationcited by 6
  11. 2025Multi-modal Summarization in Model-Based Engineering: Automotive Software Development Case Studycited by 6
  12. 2025Summarizing Speech: A Comprehensive Surveycited by 5
14 more
  1. 2022Summarizing a virtual robot's past actions in natural languagecited by 3
  2. 2024The State and Fate of Summarization Datasets: A Surveycited by 3
  3. 2022WIDAR -- Weighted Input Document Augmented ROUGEcited by 2
  4. 2023Surveying the Landscape of Text Summarization with Deep Learning: A Comprehensive Reviewcited by 2
  5. 2025Unlocking Financial Insights: An advanced Multimodal Summarization with Multimodal Output Framework for Financial Advisory Videoscited by 2
  6. 2022Grafting Pre-trained Models for Multimodal Headline Generationcited by 1
  7. 2025Multimodal Machine Learning for Real Estate Appraisal: A Comprehensive Surveycited by 1
  8. 2025Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networkscited by 1
  9. 2025MF2Summ: Multimodal Fusion for Video Summarization with Temporal Alignmentcited by 1
  10. 2025Large Language Model-Based Generation of Discharge Summariescited by 1
  11. 2023CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarizationcited by 0
  12. 2025A Novel Trustworthy Video Summarization Algorithm Through a Mixture of LoRA Expertscited by 0
  13. 2026Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attentioncited by 0
  14. 2026MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Awarecited by 0