# Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

arXiv: https://arxiv.org/abs/2010.01781 · HTML: https://mlsanity.com/paper/2010.01781/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [Sequence Level Contrastive Learning for Text Summarization](https://mlsanity.com/paper/2109.03481/) (2021)
- [A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy](https://mlsanity.com/paper/2106.13945/) (2021)
- [QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance](https://mlsanity.com/paper/2204.13921/) (2022)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization](https://mlsanity.com/paper/2305.07615/) (2023)
- [WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs](https://mlsanity.com/paper/2209.13101/) (2022)
- [Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning](https://mlsanity.com/paper/2403.05770/) (2024)
- [Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods](https://mlsanity.com/paper/2208.13265/) (2022)
- [Feeding What You Need by Understanding What You Learned](https://mlsanity.com/paper/2203.02753/) (2022)
- [UMSE: Unified Multi-scenario Summarization Evaluation](https://mlsanity.com/paper/2305.16895/) (2023)
- [Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarization](https://mlsanity.com/paper/2308.02270/) (2023)
- [Evaluation of Automatic Text Summarization using Synthetic Facts](https://mlsanity.com/paper/2204.04869/) (2022)
- [Recent Trends in Unsupervised Summarization](https://mlsanity.com/paper/2305.11231/) (2023)
- [ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks](https://mlsanity.com/paper/2603.21084/) (2026)

## Abstract (arXiv)

Evaluation of a document summarization system has been a critical factor to impact the success of the summarization task. Previous approaches, such as ROUGE, mainly consider the informativeness of the assessed summary and require human-generated references for each test summary. In this work, we propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. To learn the metric, for each summary, we construct different types of negative samples with respect to different aspects of the summary qualities, and train our model with a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries. Furthermore, we show that our method is general and transferable across datasets.