# Revisiting Grammatical Error Correction Evaluation and Beyond

arXiv: https://arxiv.org/abs/2211.01635 · HTML: https://mlsanity.com/paper/2211.01635/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020)
- [A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction](https://mlsanity.com/paper/1801.08831/) (2018)
- [On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation](https://mlsanity.com/paper/2005.01196/) (2020)
- [Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model](https://mlsanity.com/paper/2005.11849/) (2020)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning](https://mlsanity.com/paper/2012.14768/) (2020)
- [There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction](https://mlsanity.com/paper/1610.02124/) (2016)
- [USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2202.10062/) (2022)
- [Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction](https://mlsanity.com/paper/2203.13064/) (2022)
- [Classifying Syntactic Errors in Learner Language](https://mlsanity.com/paper/2010.11032/) (2020)
- [Type-Driven Multi-Turn Corrections for Grammatical Error Correction](https://mlsanity.com/paper/2203.09136/) (2022)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [KoBE: Knowledge-Based Machine Translation Evaluation](https://mlsanity.com/paper/2009.11027/) (2020)
- [Grammatical Error Correction as GAN-like Sequence Labeling](https://mlsanity.com/paper/2105.14209/) (2021)
- [Difficulty-Aware Machine Translation Evaluation](https://mlsanity.com/paper/2107.14402/) (2021)
- [BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation](https://mlsanity.com/paper/2110.09147/) (2021)
- [Variance-Aware Machine Translation Test Sets](https://mlsanity.com/paper/2111.04079/) (2021)

## Built on by (most cited first, 18 of 18)

- [ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark](https://mlsanity.com/paper/2303.13648/) (2023)
- [CoAScore: Chain-of-Aspects Prompting for NLG Evaluation](https://mlsanity.com/paper/2312.10355/) (2023)
- [Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction](https://mlsanity.com/paper/2312.08400/) (2023)
- [gec-metrics: A Unified Library for Grammatical Error Correction Evaluation](https://mlsanity.com/paper/2505.19388/) (2025)
- [Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?](https://mlsanity.com/paper/2502.09416/) (2025)
- [CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2305.10819/) (2023)
- [ChatGPT for Arabic Grammatical Error Correction](https://mlsanity.com/paper/2308.04492/) (2023)
- [Revisiting Meta-evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2403.02674/) (2024)
- [Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction](https://mlsanity.com/paper/2403.17540/) (2024)
- [IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator](https://mlsanity.com/paper/2506.02899/) (2025)
- [Leveraging What's Overfixed: Post-Correction via LLM Grammatical Error Overcorrection](https://mlsanity.com/paper/2509.20811/) (2025)
- [Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction](https://mlsanity.com/paper/2605.07635/) (2026)
- [Edit-level Majority Voting Mitigates Over-Correction in LLM-based Grammatical Error Correction](https://mlsanity.com/paper/2605.13624/) (2026)
- [Time Is Effort: Estimating Human Post-Editing Time for Grammar Error Correction Tool Evaluation](https://mlsanity.com/paper/2510.04394/) (2025)
- [JELV: A Judge of Edit-Level Validity for Evaluation and Automated Reference Expansion in Grammatical Error Correction](https://mlsanity.com/paper/2511.21700/) (2025)
- [Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation](https://mlsanity.com/paper/2602.05419/) (2026)
- [Scoring Edit Impact in Grammatical Error Correction via Embedded Association Graphs](https://mlsanity.com/paper/2604.06573/) (2026)
- [Don't Count the Edits, Judge by the Outcome Alone: Reward-Based Evaluation for Grammatical Error Correction](https://mlsanity.com/paper/2609.15559/) (2026)

## Abstract (arXiv)

Pretraining-based (PT-based) automatic evaluation metrics (e.g., BERTScore and BARTScore) have been widely used in several sentence generation tasks (e.g., machine translation and text summarization) due to their better correlation with human judgments over traditional overlap-based methods. Although PT-based methods have become the de facto standard for training grammatical error correction (GEC) systems, GEC evaluation still does not benefit from pretrained knowledge. This paper takes the first step towards understanding and improving GEC evaluation with pretraining. We first find that arbitrarily applying PT-based metrics to GEC evaluation brings unsatisfactory correlation results because of the excessive attention to inessential systems outputs (e.g., unchanged parts). To alleviate the limitation, we propose a novel GEC evaluation metric to achieve the best of both worlds, namely PT-M2 which only uses PT-based metrics to score those corrected parts. Experimental results on the CoNLL14 evaluation task show that PT-M2 significantly outperforms existing methods, achieving a new state-of-the-art result of 0.949 Pearson correlation. Further analysis reveals that PT-M2 is robust to evaluate competitive GEC systems. Source code and scripts are freely available at https://github.com/pygongnlp/PT-M2.