# Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking

arXiv: https://arxiv.org/abs/2105.12306 · HTML: https://mlsanity.com/paper/2105.12306/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Spelling Error Correction with Soft-Masked BERT](https://mlsanity.com/paper/2005.07421/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Glyce: Glyph-vectors for Chinese Character Representations](https://mlsanity.com/paper/1901.10125/) (2019)
- [Multimodal Pivots for Image Caption Translation](https://mlsanity.com/paper/1601.03916/) (2016)
- [Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets](https://mlsanity.com/paper/1704.07121/) (2017)

## Built on by (most cited first, 30 of 52)

- [The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checking](https://mlsanity.com/paper/2203.00991/) (2022)
- [On the (In)Effectiveness of Large Language Models for Chinese Text Correction](https://mlsanity.com/paper/2307.09007/) (2023)
- [Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checking](https://mlsanity.com/paper/2210.10320/) (2022)
- [General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining](https://mlsanity.com/paper/2203.10929/) (2022)
- [Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characters](https://mlsanity.com/paper/2311.11268/) (2023)
- [Improving Chinese Spelling Check by Character Pronunciation Prediction: The Effects of Adaptivity and Granularity](https://mlsanity.com/paper/2210.10996/) (2022)
- [Rethinking Masked Language Modeling for Chinese Spelling Correction](https://mlsanity.com/paper/2305.17721/) (2023)
- [A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Check](https://mlsanity.com/paper/2310.09119/) (2023)
- [Contextual Similarity is More Valuable than Character Similarity: An Empirical Study for Chinese Spell Checking](https://mlsanity.com/paper/2207.09217/) (2022)
- [Disentangled Phonetic Representation for Chinese Spelling Correction](https://mlsanity.com/paper/2305.14783/) (2023)
- [Correct Like Humans: Progressive Learning Framework for Chinese Text Error Correction](https://mlsanity.com/paper/2306.17447/) (2023)
- [C-LLM: Learn to Check Chinese Spelling Errors Character by Character](https://mlsanity.com/paper/2406.16536/) (2024)
- [Exploring and Adapting Chinese GPT to Pinyin Input Method](https://mlsanity.com/paper/2203.00249/) (2022)
- [MCSCSet: A Specialist-annotated Dataset for Medical-domain Chinese Spelling Correction](https://mlsanity.com/paper/2210.11720/) (2022)
- [Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking](https://mlsanity.com/paper/2403.08492/) (2024)
- [A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models](https://mlsanity.com/paper/2410.04027/) (2024)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [Overview of CTC 2021: Chinese Text Correction for Native Speakers](https://mlsanity.com/paper/2208.05681/) (2022)
- [Unsupervised domain adaptation for speech recognition with unsupervised error correction](https://mlsanity.com/paper/2209.12043/) (2022)
- [An Error-Guided Correction Model for Chinese Spelling Error Correction](https://mlsanity.com/paper/2301.06323/) (2023)
- [Mixture of Small and Large Models for Chinese Spelling Check](https://mlsanity.com/paper/2506.06887/) (2025)
- [SDCL: Self-Distillation Contrastive Learning for Chinese Spell Checking](https://mlsanity.com/paper/2210.17168/) (2022)
- [Error-Robust Retrieval for Chinese Spelling Check](https://mlsanity.com/paper/2211.07843/) (2022)
- [Mitigating Catastrophic Forgetting in Multi-domain Chinese Spelling Correction by Multi-stage Knowledge Transfer Framework](https://mlsanity.com/paper/2402.11422/) (2024)
- [A Training-free LLM-based Approach to General Chinese Character Error Correction](https://mlsanity.com/paper/2502.15266/) (2025)
- [A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check](https://mlsanity.com/paper/2307.13655/) (2023)
- [Does Correction Remain A Problem For Large Language Models?](https://mlsanity.com/paper/2308.01776/) (2023)
- [Enhancing conversational quality in language learning chatbots: An evaluation of GPT4 for ASR error correction](https://mlsanity.com/paper/2307.09744/) (2023)
- [RSpell: Retrieval-augmented Framework for Domain Adaptive Chinese Spelling Check](https://mlsanity.com/paper/2308.08176/) (2023)
- [Chinese Spelling Correction: A Comprehensive Survey of Progress, Challenges, and Opportunities](https://mlsanity.com/paper/2502.11508/) (2025)

## Abstract (arXiv)

Chinese Spell Checking (CSC) aims to detect and correct erroneous characters for user-generated text in the Chinese language. Most of the Chinese spelling errors are misused semantically, phonetically or graphically similar characters. Previous attempts noticed this phenomenon and try to use the similarity for this task. However, these methods use either heuristics or handcrafted confusion sets to predict the correct character. In this paper, we propose a Chinese spell checker called ReaLiSe, by directly leveraging the multimodal information of the Chinese characters. The ReaLiSe model tackles the CSC task by (1) capturing the semantic, phonetic and graphic information of the input characters, and (2) selectively mixing the information in these modalities to predict the correct output. Experiments on the SIGHAN benchmarks show that the proposed model outperforms strong baselines by a large margin.