# Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models

arXiv: https://arxiv.org/abs/2105.13318 · HTML: https://mlsanity.com/paper/2105.13318/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Tensor2Tensor for Neural Machine Translation](https://mlsanity.com/paper/1803.07416/) (2018)
- [Tagged Back-Translation](https://mlsanity.com/paper/1906.06442/) (2019)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020)
- [Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task](https://mlsanity.com/paper/1804.05940/) (2018)
- [An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction](https://mlsanity.com/paper/1909.00502/) (2019)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)
- [Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data](https://mlsanity.com/paper/1903.00138/) (2019)
- [A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction](https://mlsanity.com/paper/1801.08831/) (2018)
- [A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning](https://mlsanity.com/paper/1907.01256/) (2019)
- [Data Weighted Training Strategies for Grammatical Error Correction](https://mlsanity.com/paper/2008.02976/) (2020)
- [Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study](https://mlsanity.com/paper/1807.01270/) (2018)
- [Artificial Error Generation with Machine Translation and Syntactic Patterns](https://mlsanity.com/paper/1707.05236/) (2017)
- [Wronging a Right: Generating Better Errors to Improve Grammatical Error Detection](https://mlsanity.com/paper/1810.00668/) (2018)
- [The CUED's Grammatical Error Correction Systems for BEA-2019](https://mlsanity.com/paper/1907.00168/) (2019)

## Built on by (most cited first, 30 of 56)

- [Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation](https://mlsanity.com/paper/2304.01746/) (2023)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start](https://mlsanity.com/paper/2205.12209/) (2022)
- [Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance](https://mlsanity.com/paper/2305.13225/) (2023)
- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Prompting open-source and commercial language models for grammatical error correction of English learner text](https://mlsanity.com/paper/2401.07702/) (2024)
- [Text Generation with Text-Editing Models](https://mlsanity.com/paper/2206.07043/) (2022)
- [Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications](https://mlsanity.com/paper/2502.13358/) (2025)
- [Resilience of Large Language Models for Noisy Instructions](https://mlsanity.com/paper/2404.09754/) (2024)
- [Enhancing Grammatical Error Correction Systems with Explanations](https://mlsanity.com/paper/2305.15676/) (2023)
- [A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model](https://mlsanity.com/paper/2201.10707/) (2022)
- [Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models](https://mlsanity.com/paper/2402.02987/) (2024)
- [Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction](https://mlsanity.com/paper/2203.13064/) (2022)
- [Robustness of Large Language Models to Perturbations in Text](https://mlsanity.com/paper/2407.08989/) (2024)
- [Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models](https://mlsanity.com/paper/2204.00471/) (2022)
- [Enhancing Text Editing for Grammatical Error Correction: Arabic as a Case Study](https://mlsanity.com/paper/2503.00985/) (2025)
- [Improving Seq2Seq Grammatical Error Correction via Decoding Interventions](https://mlsanity.com/paper/2310.14534/) (2023)
- [User-Centric Gender Rewriting](https://mlsanity.com/paper/2205.02211/) (2022)
- [Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction](https://mlsanity.com/paper/2303.14342/) (2023)
- [Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora](https://mlsanity.com/paper/2305.17906/) (2023)
- [RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation](https://mlsanity.com/paper/2310.07299/) (2023)
- [Improving Grammatical Error Correction via Contextual Data Augmentation](https://mlsanity.com/paper/2406.17456/) (2024)
- [Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?](https://mlsanity.com/paper/2307.07047/) (2023)
- [Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule](https://mlsanity.com/paper/2311.11813/) (2023)
- [Adapting LLMs for Minimal-edit Grammatical Error Correction](https://mlsanity.com/paper/2506.13148/) (2025)
- [CSynGEC: Incorporating Constituent-based Syntax for Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2211.08158/) (2022)
- [LLM-based Code-Switched Text Generation for Grammatical Error Correction](https://mlsanity.com/paper/2410.10349/) (2024)
- [Diversity-Driven Combination for Grammatical Error Correction](https://mlsanity.com/paper/2110.15149/) (2021)
- [BSpell: A CNN-Blended BERT Based Bangla Spell Checker](https://mlsanity.com/paper/2208.09709/) (2022)
- [GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding](https://mlsanity.com/paper/2311.08191/) (2023)

## Abstract (arXiv)

Synthetic data generation is widely known to boost the accuracy of neural grammatical error correction (GEC) systems, but existing methods often lack diversity or are too simplistic to generate the broad range of grammatical errors made by human writers. In this work, we use error type tags from automatic annotation tools such as ERRANT to guide synthetic data generation. We compare several models that can produce an ungrammatical sentence given a clean sentence and an error type tag. We use these models to build a new, large synthetic pre-training data set with error tag frequency distributions matching a given development set. Our synthetic data set yields large and consistent gains, improving the state-of-the-art on the BEA-19 and CoNLL-14 test sets. We also show that our approach is particularly effective in adapting a GEC system, trained on mixed native and non-native English, to a native English test set, even surpassing real training data consisting of high-quality sentence pairs.