# SDCL: Self-Distillation Contrastive Learning for Chinese Spell Checking

arXiv: https://arxiv.org/abs/2210.17168 · HTML: https://mlsanity.com/paper/2210.17168/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking](https://mlsanity.com/paper/2105.12306/) (2021)
- [TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning](https://mlsanity.com/paper/2111.04198/) (2021)
- [DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning](https://mlsanity.com/paper/2104.09124/) (2021)
- [Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Models](https://mlsanity.com/paper/2105.14813/) (2021)

## Built on by (most cited first, 4 of 4)

- [Disentangled Phonetic Representation for Chinese Spelling Correction](https://mlsanity.com/paper/2305.14783/) (2023)
- [Unveiling the Impact of Multimodal Features on Chinese Spelling Correction: From Analysis to Design](https://mlsanity.com/paper/2504.07661/) (2025)
- [Investigating Glyph Phonetic Information for Chinese Spell Checking: What Works and What's Next](https://mlsanity.com/paper/2212.04068/) (2022)
- [Research on Domain-Specific Chinese Spelling Correction Method Based on Plugin Extension Modules](https://mlsanity.com/paper/2411.09884/) (2024)

## Abstract (arXiv)

Due to the ambiguity of homophones, Chinese Spell Checking (CSC) has widespread applications. Existing systems typically utilize BERT for text encoding. However, CSC requires the model to account for both phonetic and graphemic information. To adapt BERT to the CSC task, we propose a token-level self-distillation contrastive learning method. We employ BERT to encode both the corrupted and corresponding correct sentence. Then, we use contrastive learning loss to regularize corrupted tokens' hidden states to be closer to counterparts in the correct sentence. On three CSC datasets, we confirmed our method provides a significant improvement above baselines.