mlsanityData to 7 Oct 2026

Revisiting Meta-evaluation for Grammatical Error Correction

arXiv 2403.02674 ↗First posted 5 Mar 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges including biases caused by inconsistencies in evaluation granularity, and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models (LLMs), and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation, suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 72 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

26 of 72 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  7. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
  8. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  9. 2021A Simple Recipe for Multilingual Grammatical Error Correctioncited by 108
  10. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  11. 2019Parallel Iterative Edit Models for Local Sequence Transductioncited by 81
  12. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
14 more
  1. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  2. 2020Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correctioncited by 59
  3. 2019An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correctioncited by 55
  4. 2022Grammatical Error Correction: A Survey of the State of the Artcited by 43
  5. 2021LM-Critic: Language Models for Unsupervised Grammatical Error Correctioncited by 33
  6. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  7. 2016GLEU Without Tuningcited by 24
  8. 2016There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correctioncited by 18
  9. 2022Revisiting Grammatical Error Correction Evaluation and Beyondcited by 18
  10. 2018Automatic Metric Validation for Grammatical Error Correctioncited by 17
  11. 2022Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correctioncited by 12
  12. 2021Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correctioncited by 11
  13. 2018Reference-less Measure of Faithfulness for Grammatical Error Correctioncited by 9
  14. 2023Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correctioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2026Multi-Dimensional Evaluation of LLMs for Grammatical Error Correctioncited by 1
  2. 2024Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correctioncited by 0
  3. 2026Grammatical Error Correction Evaluation by Optimally Transporting Edit Representationcited by 0