# A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy

arXiv: https://arxiv.org/abs/2106.13945 · HTML: https://mlsanity.com/paper/2106.13945/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Better Rewards Yield Better Summaries: Learning to Summarise Without References](https://mlsanity.com/paper/1909.01214/) (2019)
- [SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization](https://mlsanity.com/paper/2005.03724/) (2020)
- [Sentence Centrality Revisited for Unsupervised Summarization](https://mlsanity.com/paper/1906.03508/) (2019)
- [The price of debiasing automatic metrics in natural language evaluation](https://mlsanity.com/paper/1807.02202/) (2018)
- [SumQE: a BERT-based Summary Quality Estimation Model](https://mlsanity.com/paper/1909.00578/) (2019)
- [Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning](https://mlsanity.com/paper/2010.01781/) (2020)
- [Assessing Dialogue Systems with Distribution Distances](https://mlsanity.com/paper/2105.02573/) (2021)
- [A Unified Dual-view Model for Review Summarization and Sentiment Classification with Inconsistency Loss](https://mlsanity.com/paper/2006.01592/) (2020)
- [REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation](https://mlsanity.com/paper/2105.14488/) (2021)

## Built on by (most cited first, 21 of 21)

- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation](https://mlsanity.com/paper/2310.00074/) (2023)
- [Representation Learning for Resource-Constrained Keyphrase Generation](https://mlsanity.com/paper/2203.08118/) (2022)
- [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://mlsanity.com/paper/2212.10233/) (2022)
- [Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network](https://mlsanity.com/paper/2210.05499/) (2022)
- [GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization](https://mlsanity.com/paper/2410.04087/) (2024)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation](https://mlsanity.com/paper/2306.12916/) (2023)
- [UMSE: Unified Multi-scenario Summarization Evaluation](https://mlsanity.com/paper/2305.16895/) (2023)
- [ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding](https://mlsanity.com/paper/2510.13499/) (2025)
- [LLMs as Science Journalists: Supporting Early-stage Researchers in Communicating Their Science to the Public](https://mlsanity.com/paper/2601.05821/) (2026)
- [Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarization](https://mlsanity.com/paper/2308.02270/) (2023)
- [ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answers](https://mlsanity.com/paper/2511.16846/) (2025)
- [SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder](https://mlsanity.com/paper/2207.04660/) (2022)
- [Automatic Construction of Multiple Classification Dimensions for Managing Approaches in Scientific Papers](https://mlsanity.com/paper/2505.23252/) (2025)
- [Designing and Evaluating Chain-of-Hints for Scientific Question Answering](https://mlsanity.com/paper/2510.21087/) (2025)
- [BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering](https://mlsanity.com/paper/2511.06183/) (2025)
- [COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding](https://mlsanity.com/paper/2603.21329/) (2026)

## Abstract (arXiv)

In recent years, reference-based and supervised summarization evaluation metrics have been widely explored. However, collecting human-annotated references and ratings are costly and time-consuming. To avoid these limitations, we propose a training-free and reference-free summarization evaluation metric. Our metric consists of a centrality-weighted relevance score and a self-referenced redundancy score. The relevance score is computed between the pseudo reference built from the source document and the given summary, where the pseudo reference content is weighted by the sentence centrality to provide importance guidance. Besides an F1-based relevance score, we also design an Fβ-based variant that pays more attention to the recall score. As for the redundancy score of the summary, we compute a self-masked similarity score with the summary itself to evaluate the redundant information in the summary. Finally, we combine the relevance and redundancy scores to produce the final evaluation score of the given summary. Extensive experiments show that our methods can significantly outperform existing methods on both multi-document and single-document summarization evaluation.