# There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction

arXiv: https://arxiv.org/abs/1610.02124 · HTML: https://mlsanity.com/paper/1610.02124/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2016.

## Built on by (most cited first, 18 of 18)

- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://mlsanity.com/paper/2305.14734/) (2023)
- [Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond](https://mlsanity.com/paper/2205.11484/) (2022)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [Revisiting Grammatical Error Correction Evaluation and Beyond](https://mlsanity.com/paper/2211.01635/) (2022)
- [Evaluating the Morphosyntactic Well-formedness of Generated Texts](https://mlsanity.com/paper/2103.16590/) (2021)
- [Assessing Reference-Free Peer Evaluation for Machine Translation](https://mlsanity.com/paper/2104.05146/) (2021)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2305.10819/) (2023)
- [Revisiting Meta-evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2403.02674/) (2024)
- [Introducing OmniGEC: A Silver Multilingual Dataset for Grammatical Error Correction](https://mlsanity.com/paper/2509.14504/) (2025)
- [Universal Evasion Attacks on Summarization Scoring](https://mlsanity.com/paper/2210.14260/) (2022)
- [On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems](https://mlsanity.com/paper/2104.06335/) (2021)
- [How to choose "Good" Samples for Text Data Augmentation](https://mlsanity.com/paper/2302.00894/) (2023)
- [PIE: Performance Interval Estimation for Free-Form Generation Tasks](https://mlsanity.com/paper/2509.07309/) (2025)
- [Differentially-private text generation degrades output language quality](https://mlsanity.com/paper/2509.11176/) (2025)
- [Construction of a Quality Estimation Dataset for Automatic Evaluation of Japanese Grammatical Error Correction](https://mlsanity.com/paper/2201.08038/) (2022)

## Abstract (arXiv)

Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metrics. By interpolating both methods, we achieve state-of-the-art correlation with human judgments. Finally, we show that GEC metrics are much more reliable when they are calculated at the sentence level instead of the corpus level. We have set up a CodaLab site for benchmarking GEC output using a common dataset and different evaluation metrics.