# Spelling Error Correction with Soft-Masked BERT

arXiv: https://arxiv.org/abs/2005.07421 · HTML: https://mlsanity.com/paper/2005.07421/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2020.

## Built on by (most cited first, 30 of 90)

- [The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checking](https://mlsanity.com/paper/2203.00991/) (2022)
- [On the (In)Effectiveness of Large Language Models for Chinese Text Correction](https://mlsanity.com/paper/2307.09007/) (2023)
- [Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking](https://mlsanity.com/paper/2105.12306/) (2021)
- [Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checking](https://mlsanity.com/paper/2210.10320/) (2022)
- [General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining](https://mlsanity.com/paper/2203.10929/) (2022)
- [BART based semantic correction for Mandarin automatic speech recognition system](https://mlsanity.com/paper/2104.05507/) (2021)
- [Resilience of Large Language Models for Noisy Instructions](https://mlsanity.com/paper/2404.09754/) (2024)
- [Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2106.01609/) (2021)
- [Improving Chinese Spelling Check by Character Pronunciation Prediction: The Effects of Adaptivity and Granularity](https://mlsanity.com/paper/2210.10996/) (2022)
- [Rethinking Masked Language Modeling for Chinese Spelling Correction](https://mlsanity.com/paper/2305.17721/) (2023)
- [Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Models](https://mlsanity.com/paper/2105.14813/) (2021)
- [Chinese Spelling Correction as Rephrasing Language Model](https://mlsanity.com/paper/2308.08796/) (2023)
- [A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check](https://mlsanity.com/paper/2310.09119/) (2023)
- [Contextual Similarity is More Valuable than Character Similarity: An Empirical Study for Chinese Spell Checking](https://mlsanity.com/paper/2207.09217/) (2022)
- [Disentangled Phonetic Representation for Chinese Spelling Correction](https://mlsanity.com/paper/2305.14783/) (2023)
- [CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers](https://mlsanity.com/paper/2211.08788/) (2022)
- [Correct Like Humans: Progressive Learning Framework for Chinese Text Error Correction](https://mlsanity.com/paper/2306.17447/) (2023)
- [C-LLM: Learn to Check Chinese Spelling Errors Character by Character](https://mlsanity.com/paper/2406.16536/) (2024)
- [VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning](https://mlsanity.com/paper/2112.02650/) (2021)
- [Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction](https://mlsanity.com/paper/2211.13252/) (2022)
- [CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewards](https://mlsanity.com/paper/2512.23971/) (2025)
- [uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers](https://mlsanity.com/paper/2209.07068/) (2022)
- [Effidit: Your AI Writing Assistant](https://mlsanity.com/paper/2208.01815/) (2022)
- [MCSCSet: A Specialist-annotated Dataset for Medical-domain Chinese Spelling Correction](https://mlsanity.com/paper/2210.11720/) (2022)
- [From Spelling to Grammar: A New Framework for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2211.01625/) (2022)
- [Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking](https://mlsanity.com/paper/2403.08492/) (2024)
- [A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models](https://mlsanity.com/paper/2410.04027/) (2024)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [An Error-Guided Correction Model for Chinese Spelling Error Correction](https://mlsanity.com/paper/2301.06323/) (2023)
- [Mixture of Small and Large Models for Chinese Spelling Check](https://mlsanity.com/paper/2506.06887/) (2025)

## Abstract (arXiv)

Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of-the-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using `Soft-Masked BERT' is general, and it may be employed in other language detection-correction problems. Experimental results on two datasets demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.