# CNM-BERT: A Drop-In Structural Embedding for Chinese Characters via Ideographic Description Sequences

arXiv: https://arxiv.org/abs/2608.05167 · HTML: https://mlsanity.com/paper/2608.05167/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1904.09679/) (2019)
- [ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://mlsanity.com/paper/2106.16038/) (2021)
- [DRCD: a Chinese Machine Reading Comprehension Dataset](https://mlsanity.com/paper/1806.00920/) (2018)
- [Glyce: Glyph-vectors for Chinese Character Representations](https://mlsanity.com/paper/1901.10125/) (2019)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://mlsanity.com/paper/2104.07204/) (2021)
- [A Sentence Cloze Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/2004.03116/) (2020)
- [MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition](https://mlsanity.com/paper/2107.05418/) (2021)
- [The Impact of Visual Information in Chinese Characters: Evaluating Large Models' Ability to Recognize and Utilize Radicals](https://mlsanity.com/paper/2410.09013/) (2024)
- [Radical-Enhanced Chinese Character Embedding](https://mlsanity.com/paper/1404.4714/) (2014)
- [Exploring and Adapting Chinese GPT to Pinyin Input Method](https://mlsanity.com/paper/2203.00249/) (2022)
- [Character-level Chinese-English Translation through ASCII Encoding](https://mlsanity.com/paper/1805.03330/) (2018)
- [SubCharacter Chinese-English Neural Machine Translation with Wubi encoding](https://mlsanity.com/paper/1911.02737/) (2019)

## Abstract (arXiv)

Token-based encoders like BERT treat Chinese characters as atomic identifiers, ignoring their recursive orthographic structure. Consequently, models rely on contextual co-occurrence, degrading performance on rare and out-of-vocabulary (OOV) characters. We propose the Compositional Network Model (CNM), a lightweight augmentation that injects discrete compositional structure into Transformer encoders. CNM parses Ideographic Description Sequences (IDS) into trees, encodes them via a recursive Tree-MLP, and fuses the structural embeddings into BERT without modifying the backbone. Evaluated on the Wu et al. (2025) structural-probing benchmark, CNM-BERT outperforms the strongest baseline (ChineseBERT) on long-tail and OOV characters by +9.8 Structure accuracy and +7.7 Radical F1. Furthermore, CNM-BERT achieves consistent gains across CLUE, MRC, and NER tasks at both base and large scales, demonstrating that explicit structural injection delivers both robust OOV understanding and tangible downstream value.