# Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

arXiv: https://arxiv.org/abs/1903.00138 · HTML: https://mlsanity.com/paper/1903.00138/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2019.

## Built on by (most cited first, 30 of 50)

- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [HypoGen: Hyperbole Generation with Commonsense and Counterfactual Knowledge](https://mlsanity.com/paper/2109.05097/) (2021)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://mlsanity.com/paper/2305.14734/) (2023)
- [Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2106.01609/) (2021)
- [Error Correction in ASR using Sequence-to-Sequence Models](https://mlsanity.com/paper/2202.01157/) (2022)
- [Generating Bug-Fixes Using Pretrained Transformers](https://mlsanity.com/paper/2104.07896/) (2021)
- [GPT-3.5 for Grammatical Error Correction](https://mlsanity.com/paper/2405.08469/) (2024)
- [Sequence-to-Action: Grammatical Error Correction with Action Guided Sequence Generation](https://mlsanity.com/paper/2205.10884/) (2022)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Interpretability for Language Learners Using Example-Based Grammatical Error Correction](https://mlsanity.com/paper/2203.07085/) (2022)
- [Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation](https://mlsanity.com/paper/2210.14389/) (2022)
- [Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction](https://mlsanity.com/paper/2105.04443/) (2021)
- [Comparison of Grammatical Error Correction Using Back-Translation Models](https://mlsanity.com/paper/2104.07848/) (2021)
- [uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers](https://mlsanity.com/paper/2209.07068/) (2022)
- [NaSGEC: a Multi-Domain Chinese Grammatical Error Correction Dataset from Native Speaker Texts](https://mlsanity.com/paper/2305.16023/) (2023)
- [Enhancing Text Editing for Grammatical Error Correction: Arabic as a Case Study](https://mlsanity.com/paper/2503.00985/) (2025)
- [System Combination for Grammatical Error Correction Based on Integer Programming](https://mlsanity.com/paper/2111.01465/) (2021)
- [From Spelling to Grammar: A New Framework for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2211.01625/) (2022)
- [Grammatical Error Correction as GAN-like Sequence Labeling](https://mlsanity.com/paper/2105.14209/) (2021)
- [Do Grammatical Error Correction Models Realize Grammatical Generalization?](https://mlsanity.com/paper/2106.03031/) (2021)
- [Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector](https://mlsanity.com/paper/2402.04601/) (2024)
- [A Syntax-Guided Grammatical Error Correction Model with Dependency Tree Correction](https://mlsanity.com/paper/2111.03294/) (2021)
- [A Copy Mechanism for Handling Knowledge Base Elements in SPARQL Neural Machine Translation](https://mlsanity.com/paper/2211.10271/) (2022)
- [Robust Learning for Text Classification with Multi-source Noise Simulation and Hard Example Mining](https://mlsanity.com/paper/2107.07113/) (2021)
- [Cleansing Jewel: A Neural Spelling Correction Model Built On Google OCR-ed Tibetan Manuscripts](https://mlsanity.com/paper/2304.03427/) (2023)
- [To Err Is Human, but Llamas Can Learn It Too](https://mlsanity.com/paper/2403.05493/) (2024)
- [Hierarchical Character Tagger for Short Text Spelling Error Correction](https://mlsanity.com/paper/2109.14259/) (2021)

## Abstract (arXiv)

Neural machine translation systems have become state-of-the-art approaches for Grammatical Error Correction (GEC) task. In this paper, we propose a copy-augmented architecture for the GEC task by copying the unchanged words from the source sentence to the target sentence. Since the GEC suffers from not having enough labeled training data to achieve high accuracy. We pre-train the copy-augmented architecture with a denoising auto-encoder using the unlabeled One Billion Benchmark and make comparisons between the fully pre-trained model and a partially pre-trained model. It is the first time copying words from the source context and fully pre-training a sequence to sequence model are experimented on the GEC task. Moreover, We add token-level and sentence-level multi-task learning for the GEC task. The evaluation results on the CoNLL-2014 test set show that our approach outperforms all recently published state-of-the-art results by a large margin. The code and pre-trained models are released at https://github.com/zhawe01/fairseq-gec.