# Revisiting Summarization Evaluation for Scientific Articles

arXiv: https://arxiv.org/abs/1604.00400 · HTML: https://mlsanity.com/paper/1604.00400/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2016.

## Built on by (most cited first, 30 of 31)

- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [Prompted Opinion Summarization with GPT-3.5](https://mlsanity.com/paper/2211.15914/) (2022)
- [Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias](https://mlsanity.com/paper/2401.01989/) (2024)
- [Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results](https://mlsanity.com/paper/2504.13677/) (2025)
- [Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards](https://mlsanity.com/paper/2501.07493/) (2025)
- [Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior Notes](https://mlsanity.com/paper/2104.13498/) (2021)
- [Evaluating the Efficacy of Summarization Evaluation across Languages](https://mlsanity.com/paper/2106.01478/) (2021)
- [Cut the CARP: Fishing for zero-shot story evaluation](https://mlsanity.com/paper/2110.03111/) (2021)
- [LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis](https://mlsanity.com/paper/2409.18812/) (2024)
- [CancerLLM: A Large Language Model in Cancer Domain](https://mlsanity.com/paper/2406.10459/) (2024)
- [MentSum: A Resource for Exploring Summarization of Mental Health Online Posts](https://mlsanity.com/paper/2206.00856/) (2022)
- [Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking](https://mlsanity.com/paper/2304.13854/) (2023)
- [LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systems](https://mlsanity.com/paper/2403.00982/) (2024)
- [RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine](https://mlsanity.com/paper/2410.13987/) (2024)
- [USB: A Unified Summarization Benchmark Across Tasks and Domains](https://mlsanity.com/paper/2305.14296/) (2023)
- [Large Language Models as Evaluators for Scientific Synthesis](https://mlsanity.com/paper/2407.02977/) (2024)
- [CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions](https://mlsanity.com/paper/2501.00097/) (2024)
- [TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation](https://mlsanity.com/paper/2206.00847/) (2022)
- [A comprehensive review of automatic text summarization techniques: method, data, evaluation and coding](https://mlsanity.com/paper/2301.03403/) (2023)
- [Curriculum-guided Abstractive Summarization for Mental Health Online Posts](https://mlsanity.com/paper/2302.00954/) (2023)
- [Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review](https://mlsanity.com/paper/2409.18170/) (2024)
- [Differentially Private Learning Needs Better Model Initialization and Self-Distillation](https://mlsanity.com/paper/2410.17566/) (2024)
- [Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarization](https://mlsanity.com/paper/2308.02270/) (2023)
- [The illusion of a perfect metric: Why evaluating AI's words is harder than it looks](https://mlsanity.com/paper/2508.13816/) (2025)
- [Introducing Spotlight: A Novel Approach for Generating Captivating Key Information from Documents](https://mlsanity.com/paper/2509.10935/) (2025)
- [GeSERA: General-domain Summary Evaluation by Relevance Analysis](https://mlsanity.com/paper/2110.03567/) (2021)
- [Multi-Narrative Semantic Overlap Task: Evaluation and Benchmark](https://mlsanity.com/paper/2201.05294/) (2022)
- [Curriculum-Guided Abstractive Summarization](https://mlsanity.com/paper/2302.01342/) (2023)
- [Temporal Generalization: A Reality Check](https://mlsanity.com/paper/2509.23487/) (2025)

## Abstract (arXiv)

Evaluation of text summarization approaches have been mostly based on metrics that measure similarities of system generated summaries with a set of human written gold-standard summaries. The most widely used metric in summarization evaluation has been the ROUGE family. ROUGE solely relies on lexical overlaps between the terms and phrases in the sentences; therefore, in cases of terminology variations and paraphrasing, ROUGE is not as effective. Scientific article summarization is one such case that is different from general domain summarization (e.g. newswire data). We provide an extensive analysis of ROUGE's effectiveness as an evaluation metric for scientific summarization; we show that, contrary to the common belief, ROUGE is not much reliable in evaluating scientific summaries. We furthermore show how different variants of ROUGE result in very different correlations with the manual Pyramid scores. Finally, we propose an alternative metric for summarization evaluation which is based on the content relevance between a system generated summary and the corresponding human written summaries. We call our metric SERA (Summarization Evaluation by Relevance Analysis). Unlike ROUGE, SERA consistently achieves high correlations with manual scores which shows its effectiveness in evaluation of scientific article summarization.