mlsanityData to 7 Oct 2026

Rethinking Masked Language Modeling for Chinese Spelling Correction

arXiv 2305.17721 ↗First posted 28 May 2023cs.CLcited by 19

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this paper, we study Chinese Spelling Correction (CSC) as a joint decision made by two separate models: a language model and an error model. Through empirical analysis, we find that fine-tuning BERT tends to over-fit the error model while under-fit the language model, resulting in poor generalization to out-of-distribution error patterns. Given that BERT is the backbone of most CSC models, this phenomenon has a significant negative impact. To address this issue, we are releasing a multi-domain benchmark LEMON, with higher quality and diversity than existing benchmarks, to allow a comprehensive assessment of the open domain generalization of CSC models. Then, we demonstrate that a very simple strategy, randomly masking 20% non-error tokens from the input sequence during fine-tuning is sufficient for learning a much better language model without sacrificing the error model. This technique can be applied to any model architecture and achieves new state-of-the-art results on SIGHAN, ECSpell, and LEMON.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 30 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

11 of 30 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  5. 2020Spelling Error Correction with Soft-Masked BERTcited by 90
  6. 2020SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Checkcited by 57
  7. 2021Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checkingcited by 52
  8. 2022General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretrainingcited by 25
  9. 2021Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correctioncited by 19
  10. 2021Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Modelscited by 17
  11. 2023Toward Adversarial Training on Contextualized Language Representationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 7; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characterscited by 21
  2. 2023Chinese Spelling Correction as Rephrasing Language Modelcited by 15
  3. 2024C-LLM: Learn to Check Chinese Spelling Errors Character by Charactercited by 12
  4. 2023Empower Nested Boolean Logic via Self-Supervised Curriculum Learningcited by 7
  5. 2024A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Modelscited by 7
  6. 2025Mixture of Small and Large Models for Chinese Spelling Checkcited by 5
  7. 2024Mitigating Catastrophic Forgetting in Multi-domain Chinese Spelling Correction by Multi-stage Knowledge Transfer Frameworkcited by 4
  8. 2025A Training-free LLM-based Approach to General Chinese Character Error Correctioncited by 4
  9. 2024Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learningcited by 3
  10. 2025Chinese Spelling Correction: A Comprehensive Survey of Progress, Challenges, and Opportunitiescited by 2
  11. 2025TiSpell: A Semi-Masked Methodology for Tibetan Spelling Correction covering Multi-Level Error with Data Augmentationcited by 2
  12. 2025ChineseErrorCorrector3-4B: State-of-the-Art Chinese Spelling and Grammar Correctorcited by 2
7 more
  1. 2024DISC: Plug-and-Play Decoding Intervention with Similarity of Characters for Chinese Spelling Checkcited by 1
  2. 2025Contextualized Token Discrimination for Speech Search Query Correctioncited by 1
  3. 2024Automatic Real-word Error Correction in Persian Textcited by 0
  4. 2024A Coin Has Two Sides: A Novel Detector-Corrector Framework for Chinese Spelling Correctioncited by 0
  5. 2025RAIR: Retrieval-Augmented Iterative Refinement for Chinese Spelling Correctioncited by 0
  6. 2026CLFEC: A New Task for Unified Linguistic and Factual Error Correction in paragraph-level Chinese Professional Writingcited by 0
  7. 2026GUIDE: Generative Unsupervised Chinese Query Correction via Phonetic and Visual Shared-ID Encodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023An Error-Guided Correction Model for Chinese Spelling Error Correction
  2. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewards
  3. 2025CEC-Zero: Chinese Error Correction Solution Based on LLM
  4. 2025Vision Language Models Are Not (Yet) Spelling Correctors
  5. 2023RSpell: Retrieval-augmented Framework for Domain Adaptive Chinese Spelling Check
  6. 2026CSRP: Chain-of-Thought Reasoning for Chinese Text Correction via Reinforcement Learning with Efficiency-Aware Rewards
  7. 2022Token Dropping for Efficient BERT Pretraining
  8. 2026VAA-CSEC: Vote-guided Advantage Allocation for Chinese Semantic Error Correction