# WIDAR -- Weighted Input Document Augmented ROUGE

arXiv: https://arxiv.org/abs/2201.09282 · HTML: https://mlsanity.com/paper/2201.09282/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 23 Jan 2022.

## Builds on (most cited first, 22 of 22)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [MLSUM: The Multilingual Summarization Corpus](https://mlsanity.com/paper/2004.14900/) (2020)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Better Rewards Yield Better Summaries: Learning to Summarise Without References](https://mlsanity.com/paper/1909.01214/) (2019)
- [SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization](https://mlsanity.com/paper/2005.03724/) (2020)
- [ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks](https://mlsanity.com/paper/1803.01937/) (2018)
- [Better Summarization Evaluation with Word Embeddings for ROUGE](https://mlsanity.com/paper/1508.06034/) (2015)
- [Question Answering as an Automatic Evaluation Metric for News Article Summarization](https://mlsanity.com/paper/1906.00318/) (2019)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [A Semantic QA-Based Approach for Text Summarization Evaluation](https://mlsanity.com/paper/1704.06259/) (2017)
- [Multi-Modal Summary Generation using Multi-Objective Optimization](https://mlsanity.com/paper/2005.09252/) (2020)
- [Is human scoring the best criteria for summary evaluation?](https://mlsanity.com/paper/2012.14602/) (2020)
- [Semantic Extractor-Paraphraser based Abstractive Summarization](https://mlsanity.com/paper/2105.01296/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Survey on Medical Document Summarization](https://mlsanity.com/paper/2212.01669/) (2022)
- [PerSEval: Assessing Personalization in Text Summarizers](https://mlsanity.com/paper/2407.00453/) (2024)

## Abstract (arXiv)

The task of automatic text summarization has gained a lot of traction due to the recent advancements in machine learning techniques. However, evaluating the quality of a generated summary remains to be an open problem. The literature has widely adopted Recall-Oriented Understudy for Gisting Evaluation (ROUGE) as the standard evaluation metric for summarization. However, ROUGE has some long-established limitations; a major one being its dependence on the availability of good quality reference summary. In this work, we propose the metric WIDAR which in addition to utilizing the reference summary uses also the input document in order to evaluate the quality of the generated summary. The proposed metric is versatile, since it is designed to adapt the evaluation score according to the quality of the reference summary. The proposed metric correlates better than ROUGE by 26%, 76%, 82%, and 15%, respectively, in coherence, consistency, fluency, and relevance on human judgement scores provided in the SummEval dataset. The proposed metric is able to obtain comparable results with other state-of-the-art metrics while requiring a relatively short computational time.