# A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning

arXiv: https://arxiv.org/abs/1907.01256 · HTML: https://mlsanity.com/paper/1907.01256/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2019.

## Built on by (most cited first, 26 of 26)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [Neural Inverse Text Normalization](https://mlsanity.com/paper/2102.06380/) (2021)
- [GPT-3.5 for Grammatical Error Correction](https://mlsanity.com/paper/2405.08469/) (2024)
- [Czech Grammar Error Correction with a Large and Diverse Corpus](https://mlsanity.com/paper/2201.05590/) (2022)
- [Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggers](https://mlsanity.com/paper/2402.13532/) (2024)
- [Comparison of Grammatical Error Correction Using Back-Translation Models](https://mlsanity.com/paper/2104.07848/) (2021)
- [Spelling Correction with Denoising Transformer](https://mlsanity.com/paper/2105.05977/) (2021)
- [Empirical Error Modeling Improves Robustness of Noisy Neural Sequence Labeling](https://mlsanity.com/paper/2105.11872/) (2021)
- [Understanding Model Robustness to User-generated Noisy Texts](https://mlsanity.com/paper/2110.07428/) (2021)
- [System Combination for Grammatical Error Correction Based on Integer Programming](https://mlsanity.com/paper/2111.01465/) (2021)
- [Improving Grammatical Error Correction via Contextual Data Augmentation](https://mlsanity.com/paper/2406.17456/) (2024)
- [Transfer learning for conflict and duplicate detection in software requirement pairs](https://mlsanity.com/paper/2301.03709/) (2023)
- [Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule](https://mlsanity.com/paper/2311.11813/) (2023)
- [A Syntax-Guided Grammatical Error Correction Model with Dependency Tree Correction](https://mlsanity.com/paper/2111.03294/) (2021)
- [Correcting Arabic Soft Spelling Mistakes using BiLSTM-based Machine Learning](https://mlsanity.com/paper/2108.01141/) (2021)
- [Diversity-Driven Combination for Grammatical Error Correction](https://mlsanity.com/paper/2110.15149/) (2021)
- [Cleansing Jewel: A Neural Spelling Correction Model Built On Google OCR-ed Tibetan Manuscripts](https://mlsanity.com/paper/2304.03427/) (2023)
- [PicTalky: Augmentative and Alternative Communication Software for Language Developmental Disabilities](https://mlsanity.com/paper/2109.12941/) (2021)
- [MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction](https://mlsanity.com/paper/2310.11671/) (2023)
- [System Combination via Quality Estimation for Grammatical Error Correction](https://mlsanity.com/paper/2310.14947/) (2023)
- [Detection-Correction Structure via General Language Model for Grammatical Error Correction](https://mlsanity.com/paper/2405.17804/) (2024)
- [Multi-head Sequence Tagging Model for Grammatical Error Correction](https://mlsanity.com/paper/2410.16473/) (2024)
- [Time Is Effort: Estimating Human Post-Editing Time for Grammar Error Correction Tool Evaluation](https://mlsanity.com/paper/2510.04394/) (2025)

## Abstract (arXiv)

Grammatical error correction can be viewed as a low-resource sequence-to-sequence task, because publicly available parallel corpora are limited. To tackle this challenge, we first generate erroneous versions of large unannotated corpora using a realistic noising function. The resulting parallel corpora are subsequently used to pre-train Transformer models. Then, by sequentially applying transfer learning, we adapt these models to the domain and style of the test set. Combined with a context-aware neural spellchecker, our system achieves competitive results in both restricted and low resource tracks in ACL 2019 BEA Shared Task. We release all of our code and materials for reproducibility.