# Rethinking Masked Language Modeling for Chinese Spelling Correction

arXiv: https://arxiv.org/abs/2305.17721 · HTML: https://mlsanity.com/paper/2305.17721/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 28 May 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking](https://mlsanity.com/paper/2105.12306/) (2021)
- [General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining](https://mlsanity.com/paper/2203.10929/) (2022)
- [Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2106.01609/) (2021)
- [Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Models](https://mlsanity.com/paper/2105.14813/) (2021)
- [Toward Adversarial Training on Contextualized Language Representation](https://mlsanity.com/paper/2305.04557/) (2023)

## Built on by (most cited first, 19 of 19)

- [Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characters](https://mlsanity.com/paper/2311.11268/) (2023)
- [Chinese Spelling Correction as Rephrasing Language Model](https://mlsanity.com/paper/2308.08796/) (2023)
- [C-LLM: Learn to Check Chinese Spelling Errors Character by Character](https://mlsanity.com/paper/2406.16536/) (2024)
- [Empower Nested Boolean Logic via Self-Supervised Curriculum Learning](https://mlsanity.com/paper/2310.05450/) (2023)
- [A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models](https://mlsanity.com/paper/2410.04027/) (2024)
- [Mixture of Small and Large Models for Chinese Spelling Check](https://mlsanity.com/paper/2506.06887/) (2025)
- [Mitigating Catastrophic Forgetting in Multi-domain Chinese Spelling Correction by Multi-stage Knowledge Transfer Framework](https://mlsanity.com/paper/2402.11422/) (2024)
- [A Training-free LLM-based Approach to General Chinese Character Error Correction](https://mlsanity.com/paper/2502.15266/) (2025)
- [Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning](https://mlsanity.com/paper/2411.17679/) (2024)
- [Chinese Spelling Correction: A Comprehensive Survey of Progress, Challenges, and Opportunities](https://mlsanity.com/paper/2502.11508/) (2025)
- [TiSpell: A Semi-Masked Methodology for Tibetan Spelling Correction covering Multi-Level Error with Data Augmentation](https://mlsanity.com/paper/2505.08037/) (2025)
- [ChineseErrorCorrector3-4B: State-of-the-Art Chinese Spelling and Grammar Corrector](https://mlsanity.com/paper/2511.17562/) (2025)
- [DISC: Plug-and-Play Decoding Intervention with Similarity of Characters for Chinese Spelling Check](https://mlsanity.com/paper/2412.12863/) (2024)
- [Contextualized Token Discrimination for Speech Search Query Correction](https://mlsanity.com/paper/2509.04393/) (2025)
- [Automatic Real-word Error Correction in Persian Text](https://mlsanity.com/paper/2407.14795/) (2024)
- [A Coin Has Two Sides: A Novel Detector-Corrector Framework for Chinese Spelling Correction](https://mlsanity.com/paper/2409.04150/) (2024)
- [RAIR: Retrieval-Augmented Iterative Refinement for Chinese Spelling Correction](https://mlsanity.com/paper/2504.18938/) (2025)
- [CLFEC: A New Task for Unified Linguistic and Factual Error Correction in paragraph-level Chinese Professional Writing](https://mlsanity.com/paper/2602.23845/) (2026)
- [GUIDE: Generative Unsupervised Chinese Query Correction via Phonetic and Visual Shared-ID Encoding](https://mlsanity.com/paper/2608.25343/) (2026)

## Abstract (arXiv)

In this paper, we study Chinese Spelling Correction (CSC) as a joint decision made by two separate models: a language model and an error model. Through empirical analysis, we find that fine-tuning BERT tends to over-fit the error model while under-fit the language model, resulting in poor generalization to out-of-distribution error patterns. Given that BERT is the backbone of most CSC models, this phenomenon has a significant negative impact. To address this issue, we are releasing a multi-domain benchmark LEMON, with higher quality and diversity than existing benchmarks, to allow a comprehensive assessment of the open domain generalization of CSC models. Then, we demonstrate that a very simple strategy, randomly masking 20% non-error tokens from the input sequence during fine-tuning is sufficient for learning a much better language model without sacrificing the error model. This technique can be applied to any model architecture and achieves new state-of-the-art results on SIGHAN, ECSpell, and LEMON.