# SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics

arXiv: https://arxiv.org/abs/2007.05374 · HTML: https://mlsanity.com/paper/2007.05374/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2020.

## Built on by (most cited first, 18 of 18)

- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://mlsanity.com/paper/2301.13298/) (2023)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [On the Limitations of Reference-Free Evaluations of Generated Text](https://mlsanity.com/paper/2210.12563/) (2022)
- [Finding a Balanced Degree of Automation for Summary Evaluation](https://mlsanity.com/paper/2109.11503/) (2021)
- [ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications](https://mlsanity.com/paper/2403.05303/) (2024)
- [Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations](https://mlsanity.com/paper/2305.13693/) (2023)
- [CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and Summarization](https://mlsanity.com/paper/2210.14190/) (2022)
- [NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures](https://mlsanity.com/paper/2204.13692/) (2022)
- [Pretraining Language Models Using Translationese](https://mlsanity.com/paper/2403.13638/) (2024)
- [NUTSHELL: A Dataset for Abstract Generation from Scientific Talks](https://mlsanity.com/paper/2502.16942/) (2025)
- [Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code](https://mlsanity.com/paper/2204.13848/) (2022)
- [References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation](https://mlsanity.com/paper/2506.14335/) (2025)
- [ROUGE-K: Do Your Summaries Have Keywords?](https://mlsanity.com/paper/2403.05186/) (2024)
- [KnowledgeGain: Evaluating and Optimizing Science News Generation for Reader Learning](https://mlsanity.com/paper/2605.31099/) (2026)
- [Summary Workbench: Unifying Application and Evaluation of Text Summarization Models](https://mlsanity.com/paper/2210.09587/) (2022)
- [With Good MT There is No Need For End-to-End: A Case for Translate-then-Summarize Cross-lingual Summarization](https://mlsanity.com/paper/2409.00414/) (2024)

## Abstract (arXiv)

We present SacreROUGE, an open-source library for using and developing summarization evaluation metrics. SacreROUGE removes many obstacles that researchers face when using or developing metrics: (1) The library provides Python wrappers around the official implementations of existing evaluation metrics so they share a common, easy-to-use interface; (2) it provides functionality to evaluate how well any metric implemented in the library correlates to human-annotated judgments, so no additional code needs to be written for a new evaluation metric; and (3) it includes scripts for loading datasets that contain human judgments so they can easily be used for evaluation. This work describes the design of the library, including the core Metric interface, the command-line API for evaluating summarization models and metrics, and the scripts to load and reformat publicly available datasets. The development of SacreROUGE is ongoing and open to contributions from the community.