# uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers

arXiv: https://arxiv.org/abs/2209.07068 · HTML: https://mlsanity.com/paper/2209.07068/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [Felix: Flexible Text Editing Through Tagging and Insertion](https://mlsanity.com/paper/2003.10687/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data](https://mlsanity.com/paper/1903.00138/) (2019)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study](https://mlsanity.com/paper/1807.01270/) (2018)
- [Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction](https://mlsanity.com/paper/2106.01609/) (2021)
- [A Comprehensive Survey of Grammar Error Correction](https://mlsanity.com/paper/2005.06600/) (2020)
- [Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Models](https://mlsanity.com/paper/2105.14813/) (2021)
- [Chinese Grammatical Correction Using BERT-based Pre-trained Model](https://mlsanity.com/paper/2011.02093/) (2020)
- [Grammatical Error Correction as GAN-like Sequence Labeling](https://mlsanity.com/paper/2105.14209/) (2021)
- ["Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction](https://mlsanity.com/paper/2203.00286/) (2022)

## Built on by (most cited first, 9 of 9)

- [RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models](https://mlsanity.com/paper/2406.01983/) (2024)
- [A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check](https://mlsanity.com/paper/2310.09119/) (2023)
- [DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition](https://mlsanity.com/paper/2403.17645/) (2024)
- [Contextual Spelling Correction with Language Model for Low-resource Setting](https://mlsanity.com/paper/2404.18072/) (2024)
- [Unveiling the Impact of Multimodal Features on Chinese Spelling Correction: From Analysis to Design](https://mlsanity.com/paper/2504.07661/) (2025)
- [Block the Label and Noise: An N-Gram Masked Speller for Chinese Spell Checking](https://mlsanity.com/paper/2305.03314/) (2023)
- [An Empirical Investigation of Domain Adaptation Ability for Chinese Spelling Check Models](https://mlsanity.com/paper/2401.14630/) (2024)
- [A Coin Has Two Sides: A Novel Detector-Corrector Framework for Chinese Spelling Correction](https://mlsanity.com/paper/2409.04150/) (2024)
- [GUIDE: Generative Unsupervised Chinese Query Correction via Phonetic and Visual Shared-ID Encoding](https://mlsanity.com/paper/2608.25343/) (2026)

## Abstract (arXiv)

The task of Chinese Spelling Check (CSC) is aiming to detect and correct spelling errors that can be found in the text. While manually annotating a high-quality dataset is expensive and time-consuming, thus the scale of the training dataset is usually very small (e.g., SIGHAN15 only contains 2339 samples for training), therefore supervised-learning based models usually suffer the data sparsity limitation and over-fitting issue, especially in the era of big language models. In this paper, we are dedicated to investigating the unsupervised paradigm to address the CSC problem and we propose a framework named uChecker to conduct unsupervised spelling error detection and correction. Masked pretrained language models such as BERT are introduced as the backbone model considering their powerful language diagnosis capability. Benefiting from the various and flexible MASKing operations, we propose a Confusionset-guided masking strategy to fine-train the masked language model to further improve the performance of unsupervised detection and correction. Experimental results on standard datasets demonstrate the effectiveness of our proposed model uChecker in terms of character-level and sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of spelling error detection and correction respectively.