# A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model

arXiv: https://arxiv.org/abs/2201.10707 · HTML: https://mlsanity.com/paper/2201.10707/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task](https://mlsanity.com/paper/1804.05940/) (2018)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction](https://mlsanity.com/paper/1909.00502/) (2019)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)
- [Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding](https://mlsanity.com/paper/2106.04970/) (2021)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [Unsupervised Text Generation by Learning from Search](https://mlsanity.com/paper/2007.08557/) (2020)
- [Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model](https://mlsanity.com/paper/2005.11849/) (2020)
- [Grammatical Error Correction in Low-Resource Scenarios](https://mlsanity.com/paper/1910.00353/) (2019)
- [Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study](https://mlsanity.com/paper/1807.01270/) (2018)
- [Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2006.05165/) (2020)
- [Improving Grammatical Error Correction with Machine Translation Pairs](https://mlsanity.com/paper/1911.02825/) (2019)
- [Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting](https://mlsanity.com/paper/2101.00416/) (2021)
- [Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting](https://mlsanity.com/paper/1909.06002/) (2019)
- [Chinese Grammatical Correction Using BERT-based Pre-trained Model](https://mlsanity.com/paper/2011.02093/) (2020)

## Built on by (most cited first, 16 of 16)

- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Text Revision by On-the-Fly Representation Optimization](https://mlsanity.com/paper/2204.07359/) (2022)
- [mEdIT: Multilingual Text Editing via Instruction Tuning](https://mlsanity.com/paper/2402.16472/) (2024)
- [LogiGAN: Learning Logical Reasoning via Adversarial Pre-training](https://mlsanity.com/paper/2205.08794/) (2022)
- [Deep Structural Knowledge Exploitation and Synergy for Estimating Node Importance Value on Heterogeneous Information Networks](https://mlsanity.com/paper/2402.12411/) (2024)
- [Non-autoregressive Text Editing with Copy-aware Latent Alignments](https://mlsanity.com/paper/2310.07821/) (2023)
- [Graph Component Contrastive Learning for Concept Relatedness Estimation](https://mlsanity.com/paper/2206.12556/) (2022)
- [Improving Task Generalization via Unified Schema Prompt](https://mlsanity.com/paper/2208.03229/) (2022)
- [Reducing Sequence Length by Predicting Edit Operations with Large Language Models](https://mlsanity.com/paper/2305.11862/) (2023)
- [Spivavtor: An Instruction Tuned Ukrainian Text Editing Model](https://mlsanity.com/paper/2404.18880/) (2024)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)
- [GradEscape: A Gradient-Based Evader Against AI-Generated Text Detectors](https://mlsanity.com/paper/2506.08188/) (2025)
- [Grammatical Error Correction via Mixed-Grained Weighted Training](https://mlsanity.com/paper/2311.13848/) (2023)
- [Explanation based In-Context Demonstrations Retrieval for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2502.08507/) (2025)
- [UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding](https://mlsanity.com/paper/2406.16372/) (2024)
- [Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detection](https://mlsanity.com/paper/2407.11854/) (2024)

## Abstract (arXiv)

Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns. In this paper, we propose a generic and language-independent strategy for multilingual GEC, which can train a GEC system effectively for a new non-English language with only two easy-to-access resources: 1) a pretrained cross-lingual language model (PXLM) and 2) parallel translation data between English and the language. Our approach creates diverse parallel GEC data without any language-specific operations by taking the non-autoregressive translation generated by PXLM and the gold translation as error-corrected sentence pairs. Then, we reuse PXLM to initialize the GEC model and pretrain it with the synthetic data generated by itself, which yields further improvement. We evaluate our approach on three public benchmarks of GEC in different languages. It achieves the state-of-the-art results on the NLPCC 2018 Task 2 dataset (Chinese) and obtains competitive performance on Falko-Merlin (German) and RULEC-GEC (Russian). Further analysis demonstrates that our data construction method is complementary to rule-based approaches.