# Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction

arXiv: https://arxiv.org/abs/2412.17279 · HTML: https://mlsanity.com/paper/2412.17279/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [LCSTS: A Large Scale Chinese Short Text Summarization Dataset](https://mlsanity.com/paper/1506.05865/) (2015)
- [Long and Diverse Text Generation with Planning-based Hierarchical Variational Model](https://mlsanity.com/paper/1908.06605/) (2019)
- [Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checking](https://mlsanity.com/paper/2210.10320/) (2022)
- [AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization](https://mlsanity.com/paper/2008.11869/) (2020)
- [Improving Chinese Spelling Check by Character Pronunciation Prediction: The Effects of Adaptivity and Granularity](https://mlsanity.com/paper/2210.10996/) (2022)
- [TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization](https://mlsanity.com/paper/2303.11135/) (2023)
- [Chinese Spelling Correction as Rephrasing Language Model](https://mlsanity.com/paper/2308.08796/) (2023)
- [Toward Adversarial Training on Contextualized Language Representation](https://mlsanity.com/paper/2305.04557/) (2023)
- [MultiRobustBench: Benchmarking Robustness Against Multiple Attacks](https://mlsanity.com/paper/2302.10980/) (2023)
- [Robust Fine-tuning via Perturbation and Interpolation from In-batch Instances](https://mlsanity.com/paper/2205.00633/) (2022)
- [RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation](https://mlsanity.com/paper/2310.07299/) (2023)

## Built on by (most cited first, 2 of 2)

- [C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMs](https://mlsanity.com/paper/2512.23430/) (2025)
- [RAIR: Retrieval-Augmented Iterative Refinement for Chinese Spelling Correction](https://mlsanity.com/paper/2504.18938/) (2025)

## Abstract (arXiv)

Unnatural text correction aims to automatically detect and correct spelling errors or adversarial perturbation errors in sentences. Existing methods typically rely on fine-tuning or adversarial training to correct errors, which have achieved significant success. However, these methods exhibit poor generalization performance due to the difference in data distribution between training data and real-world scenarios, known as the exposure bias problem. In this paper, we propose a self-correct adversarial training framework for LearnIng from MIsTakes (LIMIT), which is a task- and model-independent framework to correct unnatural errors or mistakes. Specifically, we fully utilize errors generated by the model that are actively exposed during the inference phase, i.e., predictions that are inconsistent with the target. This training method not only simulates potential errors in real application scenarios, but also mitigates the exposure bias of the traditional training process. Meanwhile, we design a novel decoding intervention strategy to maintain semantic consistency. Extensive experimental results on Chinese unnatural text error correction datasets show that our proposed method can correct multiple forms of errors and outperforms the state-of-the-art text correction methods. In addition, extensive results on Chinese and English datasets validate that LIMIT can serve as a plug-and-play defense module and can extend to new models and datasets without further training.