# Self-Supervised Multimodal Opinion Summarization

arXiv: https://arxiv.org/abs/2105.13135 · HTML: https://mlsanity.com/paper/2105.13135/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Localizing and Orienting Street Views Using Overhead Imagery](https://mlsanity.com/paper/1608.00161/) (2016)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://mlsanity.com/paper/1808.08858/) (2018)
- [Unsupervised Opinion Summarization as Copycat-Review Generation](https://mlsanity.com/paper/1911.02247/) (2019)
- [Sentence Centrality Revisited for Unsupervised Summarization](https://mlsanity.com/paper/1906.03508/) (2019)
- [VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles](https://mlsanity.com/paper/2010.05406/) (2020)
- [Unsupervised Opinion Summarization with Noising and Denoising](https://mlsanity.com/paper/2004.10150/) (2020)
- [Self-Supervised and Controlled Multi-Document Opinion Summarization](https://mlsanity.com/paper/2004.14754/) (2020)
- [Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting](https://mlsanity.com/paper/2005.03119/) (2020)
- [Generating Summaries with Topic Templates and Structured Convolutional Decoders](https://mlsanity.com/paper/1906.04687/) (2019)
- [Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation](https://mlsanity.com/paper/1808.09564/) (2018)
- [Unsupervised Multi-modal Neural Machine Translation](https://mlsanity.com/paper/1811.11365/) (2018)
- [Multi-modal Summarization for Video-containing Documents](https://mlsanity.com/paper/2009.08018/) (2020)

## Built on by (most cited first, 18 of 18)

- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Comparative Opinion Summarization via Collaborative Decoding](https://mlsanity.com/paper/2110.07520/) (2021)
- [One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation](https://mlsanity.com/paper/2402.11683/) (2024)
- [Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model](https://mlsanity.com/paper/2206.06606/) (2022)
- [Exploiting Pseudo Image Captions for Multimodal Summarization](https://mlsanity.com/paper/2305.05496/) (2023)
- [Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets](https://mlsanity.com/paper/2310.13340/) (2023)
- [Neural Content Extraction for Poster Generation of Scientific Papers](https://mlsanity.com/paper/2112.08550/) (2021)
- [Beyond Opinion Mining: Summarizing Opinions of Customer Reviews](https://mlsanity.com/paper/2206.01543/) (2022)
- [Product Description and QA Assisted Self-Supervised Opinion Summarization](https://mlsanity.com/paper/2404.05243/) (2024)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [Improving Radiology Summarization with Radiograph and Anatomy Prompts](https://mlsanity.com/paper/2210.08303/) (2022)
- [ICAF: Iterative Contrastive Alignment Framework for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2108.05123/) (2021)
- [Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarization](https://mlsanity.com/paper/2208.11303/) (2022)
- [Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization](https://mlsanity.com/paper/2402.15473/) (2024)
- [Distilling Opinions at Scale: Incremental Opinion Summarization using XL-OPSUMM](https://mlsanity.com/paper/2406.10886/) (2024)
- [AaKOS: Aspect-adaptive Knowledge-based Opinion Summarization](https://mlsanity.com/paper/2306.05537/) (2023)
- ["This Suits You the Best": Query Focused Comparative Explainable Summarization](https://mlsanity.com/paper/2507.04733/) (2025)
- [Less Is More? Selective Visual Attention to High-Importance Regions for Multimodal Radiology Summarization](https://mlsanity.com/paper/2603.29901/) (2026)

## Abstract (arXiv)

Recently, opinion summarization, which is the generation of a summary from multiple reviews, has been conducted in a self-supervised manner by considering a sampled review as a pseudo summary. However, non-text data such as image and metadata related to reviews have been considered less often. To use the abundant information contained in non-text data, we propose a self-supervised multimodal opinion summarization framework called MultimodalSum. Our framework obtains a representation of each modality using a separate encoder for each modality, and the text decoder generates a summary. To resolve the inherent heterogeneity of multimodal data, we propose a multimodal training pipeline. We first pretrain the text encoder--decoder based solely on text modality data. Subsequently, we pretrain the non-text modality encoders by considering the pretrained text decoder as a pivot for the homogeneous representation of multimodal data. Finally, to fuse multimodal representations, we train the entire framework in an end-to-end manner. We demonstrate the superiority of MultimodalSum by conducting experiments on Yelp and Amazon datasets.