# Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting

arXiv: https://arxiv.org/abs/2310.13321 · HTML: https://mlsanity.com/paper/2310.13321/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 100 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Encode, Tag, Realize: High-Precision Text Editing](https://mlsanity.com/paper/1909.01187/) (2019)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [Towards Robustness Against Natural Language Word Substitutions](https://mlsanity.com/paper/2107.13541/) (2021)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020)
- [Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task](https://mlsanity.com/paper/1804.05940/) (2018)
- [Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution](https://mlsanity.com/paper/2108.12777/) (2021)
- [An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction](https://mlsanity.com/paper/1909.00502/) (2019)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)

## Built on by (most cited first, 2 of 2)

- [RoLegalGEC: Legal Domain Grammatical Error Detection and Correction Dataset for Romanian](https://mlsanity.com/paper/2604.19593/) (2026)
- [CoCoGEC: Counterfactual Generation for Robust Grammatical Error Correction](https://mlsanity.com/paper/2606.15069/) (2026)

## Abstract (arXiv)

Recent studies have revealed that grammatical error correction methods in the sequence-to-sequence paradigm are vulnerable to adversarial attack, and simply utilizing adversarial examples in the pre-training or post-training process can significantly enhance the robustness of GEC models to certain types of attack without suffering too much performance loss on clean data. In this paper, we further conduct a thorough robustness evaluation of cutting-edge GEC methods for four different types of adversarial attacks and propose a simple yet very effective Cycle Self-Augmenting (CSA) method accordingly. By leveraging the augmenting data from the GEC models themselves in the post-training process and introducing regularization data for cycle training, our proposed method can effectively improve the model robustness of well-trained GEC models with only a few more training epochs as an extra cost. More concretely, further training on the regularization data can prevent the GEC models from over-fitting on easy-to-learn samples and thus can improve the generalization capability and robustness towards unseen data (adversarial noise/samples). Meanwhile, the self-augmented data can provide more high-quality pseudo pairs to improve model performance on the original testing data. Experiments on four benchmark datasets and seven strong models indicate that our proposed training method can significantly enhance the robustness of four types of attacks without using purposely built adversarial examples in training. Evaluation results on clean data further confirm that our proposed CSA method significantly improves the performance of four baselines and yields nearly comparable results with other state-of-the-art models. Our code is available at https://github.com/ZetangForward/CSA-GEC.