# Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule

arXiv: https://arxiv.org/abs/2311.11813 · HTML: https://mlsanity.com/paper/2311.11813/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation](https://mlsanity.com/paper/2304.01746/) (2023)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://mlsanity.com/paper/2105.13318/) (2021)
- [An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction](https://mlsanity.com/paper/1909.00502/) (2019)
- [Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model](https://mlsanity.com/paper/2005.11849/) (2020)
- [SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser](https://mlsanity.com/paper/2210.12484/) (2022)
- [Grammatical Error Correction in Low-Resource Scenarios](https://mlsanity.com/paper/1910.00353/) (2019)
- [Text Simplification by Tagging](https://mlsanity.com/paper/2103.05070/) (2021)
- [A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning](https://mlsanity.com/paper/1907.01256/) (2019)
- [Data Weighted Training Strategies for Grammatical Error Correction](https://mlsanity.com/paper/2008.02976/) (2020)
- [Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting](https://mlsanity.com/paper/1909.06002/) (2019)
- [Type-Driven Multi-Turn Corrections for Grammatical Error Correction](https://mlsanity.com/paper/2203.09136/) (2022)
- [The Unbearable Weight of Generating Artificial Errors for Grammatical Error Correction](https://mlsanity.com/paper/1907.08889/) (2019)
- [A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction](https://mlsanity.com/paper/2010.03155/) (2020)
- [GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding](https://mlsanity.com/paper/2311.08191/) (2023)

## Built on by (most cited first, 5 of 5)

- [Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications](https://mlsanity.com/paper/2502.13358/) (2025)
- [Robustness of Large Language Models to Perturbations in Text](https://mlsanity.com/paper/2407.08989/) (2024)
- [Improving Grammatical Error Correction via Contextual Data Augmentation](https://mlsanity.com/paper/2406.17456/) (2024)
- [Efficient and Interpretable Grammatical Error Correction with Mixture of Experts](https://mlsanity.com/paper/2410.23507/) (2024)
- [Detection-Correction Structure via General Language Model for Grammatical Error Correction](https://mlsanity.com/paper/2405.17804/) (2024)

## Abstract (arXiv)

Progress in neural grammatical error correction (GEC) is hindered by the lack of annotated training data. Sufficient amounts of high-quality manually annotated data are not available, so recent research has relied on generating synthetic data, pretraining on it, and then fine-tuning on real datasets; performance gains have been achieved either by ensembling or by using huge pretrained models such as XXL-T5 as the backbone. In this work, we explore an orthogonal direction: how to use available data more efficiently. First, we propose auxiliary tasks that exploit the alignment between the original and corrected sentences, such as predicting a sequence of corrections. We formulate each task as a sequence-to-sequence problem and perform multi-task training. Second, we discover that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance, so we set out to find the best training schedule. Together, these two ideas lead to significant improvements, producing results that improve state of the art with much smaller models; in particular, we outperform the best models based on T5-XXL (11B parameters) with a BART-based model (400M parameters).