# Revisiting Meta-evaluation for Grammatical Error Correction

arXiv: https://arxiv.org/abs/2403.02674 · HTML: https://mlsanity.com/paper/2403.02674/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 72 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization](https://mlsanity.com/paper/2311.09184/) (2023)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction](https://mlsanity.com/paper/1909.00502/) (2019)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [Benchmarking Large Language Model Capabilities for Conditional Generation](https://mlsanity.com/paper/2306.16793/) (2023)
- [GLEU Without Tuning](https://mlsanity.com/paper/1605.02592/) (2016)
- [There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction](https://mlsanity.com/paper/1610.02124/) (2016)
- [Revisiting Grammatical Error Correction Evaluation and Beyond](https://mlsanity.com/paper/2211.01635/) (2022)
- [Automatic Metric Validation for Grammatical Error Correction](https://mlsanity.com/paper/1804.11225/) (2018)
- [Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction](https://mlsanity.com/paper/2203.13064/) (2022)
- [Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction](https://mlsanity.com/paper/2105.04443/) (2021)
- [Reference-less Measure of Faithfulness for Grammatical Error Correction](https://mlsanity.com/paper/1804.03824/) (2018)
- [Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction](https://mlsanity.com/paper/2303.14342/) (2023)

## Built on by (most cited first, 3 of 3)

- [Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction](https://mlsanity.com/paper/2605.07635/) (2026)
- [Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction](https://mlsanity.com/paper/2412.13110/) (2024)
- [Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation](https://mlsanity.com/paper/2602.05419/) (2026)

## Abstract (arXiv)

Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges including biases caused by inconsistencies in evaluation granularity, and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models (LLMs), and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation, suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.