# Facet-Aware Evaluation for Extractive Summarization

arXiv: https://arxiv.org/abs/1908.10383 · HTML: https://mlsanity.com/paper/1908.10383/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2019.

## Built on by (most cited first, 5 of 5)

- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision](https://mlsanity.com/paper/2205.06207/) (2022)
- [An Efficient Coarse-to-Fine Facet-Aware Unsupervised Summarization Framework based on Semantic Blocks](https://mlsanity.com/paper/2208.08253/) (2022)
- [Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation](https://mlsanity.com/paper/2104.08724/) (2021)
- [Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarization](https://mlsanity.com/paper/2308.02270/) (2023)

## Abstract (arXiv)

Commonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage in extracted summaries. Specifically, we treat each sentence in the reference summary as a textitfacet, identify the sentences in the document that express the semantics of each facet as textitsupport sentences of the facet, and automatically evaluate extractive summarization methods by comparing the indices of extracted sentences and support sentences of all the facets in the reference summary. To facilitate this new evaluation setup, we construct an extractive version of the CNN/Daily Mail dataset and perform a thorough quantitative investigation, through which we demonstrate that facet-aware evaluation manifests better correlation with human judgment than ROUGE, enables fine-grained evaluation as well as comparative analysis, and reveals valuable insights of state-of-the-art summarization methods. Data can be found at https://github.com/morningmoni/FAR.