# Automatic Metric Validation for Grammatical Error Correction

arXiv: https://arxiv.org/abs/1804.11225 · HTML: https://mlsanity.com/paper/1804.11225/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2018.

## Built on by (most cited first, 17 of 17)

- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [EditEval: An Instruction-Based Benchmark for Text Improvements](https://mlsanity.com/paper/2209.13331/) (2022)
- [CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions](https://mlsanity.com/paper/2403.00241/) (2024)
- [Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation](https://mlsanity.com/paper/2210.14389/) (2022)
- [SERRANT: a syntactic classifier for English Grammatical Error Types](https://mlsanity.com/paper/2104.02310/) (2021)
- [CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2305.10819/) (2023)
- [ComSum: Commit Messages Summarization and Meaning Preservation](https://mlsanity.com/paper/2108.10763/) (2021)
- [Revisiting Meta-evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2403.02674/) (2024)
- [IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator](https://mlsanity.com/paper/2506.02899/) (2025)
- [Some Grammatical Errors are Frequent, Others are Important](https://mlsanity.com/paper/2205.05730/) (2022)
- [MuLER: Detailed and Scalable Reference-based Evaluation](https://mlsanity.com/paper/2305.14991/) (2023)
- [JELV: A Judge of Edit-Level Validity for Evaluation and Automated Reference Expansion in Grammatical Error Correction](https://mlsanity.com/paper/2511.21700/) (2025)
- [Minimal-Edit Instruction Tuning for Low-Resource Indic GEC](https://mlsanity.com/paper/2512.00219/) (2025)
- [Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures](https://mlsanity.com/paper/2609.10893/) (2026)

## Abstract (arXiv)

Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with maege shed a new light on metric quality, showing for example that the standard M2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.