# Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training

arXiv: https://arxiv.org/abs/2305.18760 · HTML: https://mlsanity.com/paper/2305.18760/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 57 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](https://mlsanity.com/paper/2201.08860/) (2022)
- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022)
- [JAKET: Joint Pre-training of Knowledge Graph and Language Understanding](https://mlsanity.com/paper/2010.00796/) (2020)
- [ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://mlsanity.com/paper/2106.16038/) (2021)
- [Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data](https://mlsanity.com/paper/2203.08773/) (2022)
- [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://mlsanity.com/paper/2012.00413/) (2020)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [Do Syntax Trees Help Pre-trained Transformers Extract Information?](https://mlsanity.com/paper/2008.09084/) (2020)
- [Glyce: Glyph-vectors for Chinese Character Representations](https://mlsanity.com/paper/1901.10125/) (2019)
- [JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering](https://mlsanity.com/paper/2112.02732/) (2021)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Kformer: Knowledge Injection in Transformer Feed-Forward Layers](https://mlsanity.com/paper/2201.05742/) (2022)
- [Improving BERT with Syntax-aware Local Attention](https://mlsanity.com/paper/2012.15150/) (2020)

## Built on by (most cited first, 1 of 1)

- [Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry](https://mlsanity.com/paper/2505.13210/) (2025)

## Abstract (arXiv)

We introduce CDBERT, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters. We name the two core modules of CDBERT as Shuowen and Jiezi, where Shuowen refers to the process of retrieving the most appropriate meaning from Chinese dictionaries and Jiezi refers to the process of enhancing characters' glyph representations with structure understanding. To facilitate dictionary understanding, we propose three pre-training tasks, i.e., Masked Entry Modeling, Contrastive Learning for Synonym and Antonym, and Example Learning. We evaluate our method on both modern Chinese understanding benchmark CLUE and ancient Chinese benchmark CCLUE. Moreover, we propose a new polysemy discrimination task PolyMRC based on the collected dictionary of ancient Chinese. Our paradigm demonstrates consistent improvements on previous Chinese PLMs across all tasks. Moreover, our approach yields significant boosting on few-shot setting of ancient Chinese understanding.