# Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models --- Is Single-Corpus Evaluation Enough?

arXiv: https://arxiv.org/abs/1904.02927 · HTML: https://mlsanity.com/paper/1904.02927/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 6 of 6)

- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2204.10994/) (2022)
- [Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond](https://mlsanity.com/paper/2205.11484/) (2022)
- [Do Grammatical Error Correction Models Realize Grammatical Generalization?](https://mlsanity.com/paper/2106.03031/) (2021)
- [Proficiency Matters Quality Estimation in Grammatical Error Correction](https://mlsanity.com/paper/2201.06199/) (2022)

## Abstract (arXiv)

This study explores the necessity of performing cross-corpora evaluation for grammatical error correction (GEC) models. GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark. However, the evaluation remains incomplete because the task difficulty varies depending on the test corpus and conditions such as the proficiency levels of the writers and essay topics. To overcome this limitation, we evaluate the performance of several GEC models, including NMT-based (LSTM, CNN, and transformer) and an SMT-based model, against various learner corpora (CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal that the models' rankings considerably vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models.