mlsanityData to 14 Oct 2026

OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization

arXiv 2310.18122 ↗First posted 27 Oct 2023cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Opinion summarization sets itself apart from other types of summarization tasks due to its distinctive focus on aspects and sentiments. Although certain automated evaluation methods like ROUGE have gained popularity, we have found them to be unreliable measures for assessing the quality of opinion summaries. In this paper, we present OpinSummEval, a dataset comprising human judgments and outputs from 14 opinion summarization models. We further explore the correlation between 24 automatic metrics and human ratings across four dimensions. Our findings indicate that metrics based on neural networks generally outperform non-neural ones. However, even metrics built on powerful backbones, such as BART and GPT-3/3.5, do not consistently correlate well across all dimensions, highlighting the need for advancements in automated evaluation methods for opinion summarization. The code and data are publicly available at https://github.com/A-Chicharito-S/OpinSummEval/tree/main.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 36 of its 78 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

36 of 78 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
24 more
  1. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  2. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  3. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  4. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  5. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  6. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  7. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  8. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  9. 2018Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervisedcited by 75
  10. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  11. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  12. 2019Unsupervised Opinion Summarization as Copycat-Review Generationcited by 68
  13. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  14. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  15. 2020Unsupervised Opinion Summarization with Noising and Denoisingcited by 37
  16. 2020OpinionDigest: A Simple Framework for Opinion Summarizationcited by 37
  17. 2019Informative and Controllable Opinion Summarizationcited by 27
  18. 2020Unsupervised Opinion Summarization with Content Planningcited by 27
  19. 2021Learning Opinion Summarizers by Selecting Informative Reviewscited by 25
  20. 2022Efficient Few-Shot Fine-Tuning for Opinion Summarizationcited by 16
  21. 2021Unsupervised Abstractive Opinion Summarization by Generating Sentences with Tree-Structured Topic Guidancecited by 14
  22. 2019Unsupervised Neural Single-Document Summarization of Reviews via Learning Latent Discourse Structure and its Rankingcited by 11
  23. 2019Weakly-Supervised Opinion Summarization by Leveraging External Informationcited by 11
  24. 2023Revisiting Automatic Question Summarization Evaluation in the Biomedical Domaincited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 2; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  3. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  4. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  5. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  6. 2024One Prompt To Rule Them All: LLMs for Opinion Summary Evaluationcited by 16
  7. 2024Hierarchical Indexing for Retrieval-Augmented Opinion Summarizationcited by 7
  8. 2025OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviewscited by 4
  9. 2024LFOSum: Summarizing Long-form Opinions with Large Language Modelscited by 1
  10. 2025LLMs as Architects and Critics for Multi-Source Opinion Summarizationcited by 0
  11. 2026FactSim: Fact-Checking for Opinion Summarizationcited by 0
  12. 2026ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctnesscited by 0
1 more
  1. 2026Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Textcited by 0