# Automatically Evaluating Opinion Prevalence in Opinion Summarization

arXiv: https://arxiv.org/abs/2307.14305 · HTML: https://mlsanity.com/paper/2307.14305/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://mlsanity.com/paper/2005.00481/) (2020)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://mlsanity.com/paper/1808.08858/) (2018)
- [Unsupervised Opinion Summarization as Copycat-Review Generation](https://mlsanity.com/paper/1911.02247/) (2019)
- [Unsupervised Opinion Summarization with Noising and Denoising](https://mlsanity.com/paper/2004.10150/) (2020)
- [OpinionDigest: A Simple Framework for Opinion Summarization](https://mlsanity.com/paper/2005.01901/) (2020)
- [Learning Opinion Summarizers by Selecting Informative Reviews](https://mlsanity.com/paper/2109.04325/) (2021)
- [Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline](https://mlsanity.com/paper/2009.00590/) (2020)
- [Convex Aggregation for Opinion Summarization](https://mlsanity.com/paper/2104.01371/) (2021)
- [Unsupervised Abstractive Opinion Summarization by Generating Sentences with Tree-Structured Topic Guidance](https://mlsanity.com/paper/2106.08007/) (2021)
- [Beyond Opinion Mining: Summarizing Opinions of Customer Reviews](https://mlsanity.com/paper/2206.01543/) (2022)
- [Attributable and Scalable Opinion Summarization](https://mlsanity.com/paper/2305.11603/) (2023)
- [Noisy Pairing and Partial Supervision for Stylized Opinion Summarization](https://mlsanity.com/paper/2211.08723/) (2022)

## Built on by (most cited first, 1 of 1)

- [Hierarchical Indexing for Retrieval-Augmented Opinion Summarization](https://mlsanity.com/paper/2403.00435/) (2024)

## Abstract (arXiv)

When faced with a large number of product reviews, it is not clear that a human can remember all of them and weight opinions representatively to write a good reference summary. We propose an automatic metric to test the prevalence of the opinions that a summary expresses, based on counting the number of reviews that are consistent with each statement in the summary, while discrediting trivial or redundant statements. To formulate this opinion prevalence metric, we consider several existing methods to score the factual consistency of a summary statement with respect to each individual source review. On a corpus of Amazon product reviews, we gather multiple human judgments of the opinion consistency, to determine which automatic metric best expresses consistency in product reviews. Using the resulting opinion prevalence metric, we show that a human authored summary has only slightly better opinion prevalence than randomly selected extracts from the source reviews, and previous extractive and abstractive unsupervised opinion summarization methods perform worse than humans. We demonstrate room for improvement with a greedy construction of extractive summaries with twice the opinion prevalence achieved by humans. Finally, we show that preprocessing source reviews by simplification can raise the opinion prevalence achieved by existing abstractive opinion summarization systems to the level of human performance.