# Corpora Generation for Grammatical Error Correction

arXiv: https://arxiv.org/abs/1904.05780 · HTML: https://mlsanity.com/paper/1904.05780/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2019.

## Built on by (most cited first, 30 of 53)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://mlsanity.com/paper/2305.08844/) (2023)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding](https://mlsanity.com/paper/2106.04970/) (2021)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Generate, Annotate, and Learn: NLP with Synthetic Text](https://mlsanity.com/paper/2106.06168/) (2021)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Quiz-Style Question Generation for News Stories](https://mlsanity.com/paper/2102.09094/) (2021)
- [Czech Grammar Error Correction with a Large and Diverse Corpus](https://mlsanity.com/paper/2201.05590/) (2022)
- [A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model](https://mlsanity.com/paper/2201.10707/) (2022)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [On Robustness of Neural Semantic Parsers](https://mlsanity.com/paper/2102.01563/) (2021)
- [Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model](https://mlsanity.com/paper/2305.11140/) (2023)
- [Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation](https://mlsanity.com/paper/2210.14389/) (2022)
- [Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction](https://mlsanity.com/paper/2203.13064/) (2022)
- [Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction](https://mlsanity.com/paper/2105.04443/) (2021)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [Comparison of Grammatical Error Correction Using Back-Translation Models](https://mlsanity.com/paper/2104.07848/) (2021)
- [Type-Driven Multi-Turn Corrections for Grammatical Error Correction](https://mlsanity.com/paper/2203.09136/) (2022)
- [Empirical Error Modeling Improves Robustness of Noisy Neural Sequence Labeling](https://mlsanity.com/paper/2105.11872/) (2021)
- [To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support](https://mlsanity.com/paper/2305.16799/) (2023)
- [From Spelling to Grammar: A New Framework for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2211.01625/) (2022)
- [Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora](https://mlsanity.com/paper/2305.17906/) (2023)
- [Grammatical Error Correction as GAN-like Sequence Labeling](https://mlsanity.com/paper/2105.14209/) (2021)
- [Improving Grammatical Error Correction via Contextual Data Augmentation](https://mlsanity.com/paper/2406.17456/) (2024)

## Abstract (arXiv)

Grammatical Error Correction (GEC) has been recently modeled using the sequence-to-sequence framework. However, unlike sequence transduction problems such as machine translation, GEC suffers from the lack of plentiful parallel data. We describe two approaches for generating large parallel datasets for GEC using publicly available Wikipedia data. The first method extracts source-target pairs from Wikipedia edit histories with minimal filtration heuristics, while the second method introduces noise into Wikipedia sentences via round-trip translation through bridge languages. Both strategies yield similar sized parallel corpora containing around 4B tokens. We employ an iterative decoding strategy that is tailored to the loosely supervised nature of our constructed corpora. We demonstrate that neural GEC models trained using either type of corpora give similar performance. Fine-tuning these models on the Lang-8 corpus and ensembling allows us to surpass the state of the art on both the CoNLL-2014 benchmark and the JFLEG task. We provide systematic analysis that compares the two approaches to data generation and highlights the effectiveness of ensembling.