# One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation

arXiv: https://arxiv.org/abs/2402.11683 · HTML: https://mlsanity.com/paper/2402.11683/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [Unsupervised Opinion Summarization as Copycat-Review Generation](https://mlsanity.com/paper/1911.02247/) (2019)
- [Neural Network-Based Abstract Generation for Opinions and Arguments](https://mlsanity.com/paper/1606.02785/) (2016)
- [Unsupervised Opinion Summarization with Noising and Denoising](https://mlsanity.com/paper/2004.10150/) (2020)
- [Improving Abstraction in Text Summarization](https://mlsanity.com/paper/1808.07913/) (2018)
- [Unsupervised Opinion Summarization with Content Planning](https://mlsanity.com/paper/2012.07808/) (2020)
- [Self-Supervised and Controlled Multi-Document Opinion Summarization](https://mlsanity.com/paper/2004.14754/) (2020)
- [Self-Supervised Multimodal Opinion Summarization](https://mlsanity.com/paper/2105.13135/) (2021)
- [OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization](https://mlsanity.com/paper/2310.18122/) (2023)

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education](https://mlsanity.com/paper/2404.06711/) (2024)
- [Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions](https://mlsanity.com/paper/2408.08781/) (2024)
- [HPSS: Heuristic Prompting Strategy Search for LLM Evaluators](https://mlsanity.com/paper/2502.13031/) (2025)
- [OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews](https://mlsanity.com/paper/2509.00285/) (2025)
- [Can Large Language Models be Effective Online Opinion Miners?](https://mlsanity.com/paper/2505.15695/) (2025)
- [Rehearse With User: Personalized Opinion Summarization via Role-Playing based on Large Language Models](https://mlsanity.com/paper/2503.00449/) (2025)
- [LFOSum: Summarizing Long-form Opinions with Large Language Models](https://mlsanity.com/paper/2410.13037/) (2024)
- [A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretability](https://mlsanity.com/paper/2502.12052/) (2025)
- [User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios](https://mlsanity.com/paper/2510.20721/) (2025)
- [See Where You Read with Eye Gaze Tracking and Large Language Model](https://mlsanity.com/paper/2409.19454/) (2024)
- ["This Suits You the Best": Query Focused Comparative Explainable Summarization](https://mlsanity.com/paper/2507.04733/) (2025)
- [Nyay-Darpan: Enhancing Decision Making Through Summarization and Case Retrieval for Consumer Law in India](https://mlsanity.com/paper/2507.06090/) (2025)
- [DiSCo: Making Absence Visible in Intelligent Summarization Interfaces](https://mlsanity.com/paper/2601.07229/) (2026)
- [Leveraging Weighted Syntactic and Semantic Context Assessment Summary (wSSAS) Towards Text Categorization Using LLMs](https://mlsanity.com/paper/2604.12049/) (2026)
- [Consistency Analysis of Sentiment Predictions using Syntactic & Semantic Context Assessment Summarization (SSAS)](https://mlsanity.com/paper/2604.15547/) (2026)

## Abstract (arXiv)

Evaluation of opinion summaries using conventional reference-based metrics rarely provides a holistic evaluation and has been shown to have a relatively low correlation with human judgments. Recent studies suggest using Large Language Models (LLMs) as reference-free metrics for NLG evaluation, however, they remain unexplored for opinion summary evaluation. Moreover, limited opinion summary evaluation datasets inhibit progress. To address this, we release the SUMMEVAL-OP dataset covering 7 dimensions related to the evaluation of opinion summaries: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity. We investigate Op-I-Prompt a dimension-independent prompt, and Op-Prompts, a dimension-dependent set of prompts for opinion summary evaluation. Experiments indicate that Op-I-Prompt emerges as a good alternative for evaluating opinion summaries achieving an average Spearman correlation of 0.70 with humans, outperforming all previous approaches. To the best of our knowledge, we are the first to investigate LLMs as evaluators on both closed-source and open-source models in the opinion summarization domain.