# Inherent Biases in Reference based Evaluation for Grammatical Error Correction and Text Simplification

arXiv: https://arxiv.org/abs/1804.11254 · HTML: https://mlsanity.com/paper/1804.11254/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2018.

## Built on by (most cited first, 15 of 15)

- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [The Grammar-Learning Trajectories of Neural Language Models](https://mlsanity.com/paper/2109.06096/) (2021)
- [MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2204.10994/) (2022)
- [A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents](https://mlsanity.com/paper/2305.14772/) (2023)
- [Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA](https://mlsanity.com/paper/2305.14458/) (2023)
- [PreQuEL: Quality Estimation of Machine Translation Outputs in Advance](https://mlsanity.com/paper/2205.09178/) (2022)
- [BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation](https://mlsanity.com/paper/2110.09147/) (2021)
- [ComSum: Commit Messages Summarization and Meaning Preservation](https://mlsanity.com/paper/2108.10763/) (2021)
- [Focus Is What You Need For Chinese Grammatical Error Correction](https://mlsanity.com/paper/2210.12692/) (2022)
- [IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator](https://mlsanity.com/paper/2506.02899/) (2025)
- [Proficiency Matters Quality Estimation in Grammatical Error Correction](https://mlsanity.com/paper/2201.06199/) (2022)
- [Some Grammatical Errors are Frequent, Others are Important](https://mlsanity.com/paper/2205.05730/) (2022)
- [Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation](https://mlsanity.com/paper/2602.05419/) (2026)
- [Listen, Correct, and Feed Back: Spoken Pedagogical Feedback Generation](https://mlsanity.com/paper/2604.14177/) (2026)

## Abstract (arXiv)

The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality (it low coverage bias or LCB). This paper shows that overcoming LCB in Grammatical Error Correction (GEC) evaluation cannot be attained by re-scaling or by increasing the number of references in any feasible range, contrary to previous suggestions. This is due to the long-tailed distribution of valid corrections for a sentence. Concretely, we show that LCB incentivizes GEC systems to avoid correcting even when they can generate a valid correction. Consequently, existing systems obtain comparable or superior performance compared to humans, by making few but targeted changes to the input. Similar effects on Text Simplification further support our claims.