# Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

arXiv: https://arxiv.org/abs/2104.07204 · HTML: https://mlsanity.com/paper/2104.07204/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [FLAT: Chinese NER Using Flat-Lattice Transformer](https://mlsanity.com/paper/2004.11795/) (2020)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization](https://mlsanity.com/paper/2008.11869/) (2020)
- [LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching](https://mlsanity.com/paper/2102.12671/) (2021)
- [Neural Lattice Language Models](https://mlsanity.com/paper/1803.05071/) (2018)
- [Nugget Proposal Networks for Chinese Event Detection](https://mlsanity.com/paper/1805.00249/) (2018)
- [Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention](https://mlsanity.com/paper/1911.02821/) (2019)
- [Lattice CNNs for Matching Based Chinese Question Answering](https://mlsanity.com/paper/1902.09087/) (2019)

## Built on by (most cited first, 19 of 19)

- [SuperBPE: Space Travel for Language Models](https://mlsanity.com/paper/2503.13423/) (2025)
- [Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understanding](https://mlsanity.com/paper/2206.13155/) (2022)
- [Sub-Character Tokenization for Chinese Pretrained Language Models](https://mlsanity.com/paper/2106.00400/) (2021)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)
- ["Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction](https://mlsanity.com/paper/2203.00286/) (2022)
- [EEL: Efficiently Encoding Lattices for Reranking](https://mlsanity.com/paper/2306.00947/) (2023)
- [MarkBERT: Marking Word Boundaries Improves Chinese BERT](https://mlsanity.com/paper/2203.06378/) (2022)
- [CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations](https://mlsanity.com/paper/2208.10844/) (2022)
- [LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloud](https://mlsanity.com/paper/2309.17157/) (2023)
- [Enhance Robustness of Language Models Against Variation Attack through Graph Integration](https://mlsanity.com/paper/2404.12014/) (2024)
- [Improving Large Language Models with Concept-Aware Fine-Tuning](https://mlsanity.com/paper/2506.07833/) (2025)
- [An In-depth Study on Internal Structure of Chinese Words](https://mlsanity.com/paper/2106.00334/) (2021)
- [Dynamic Multi-View Fusion Mechanism For Chinese Relation Extraction](https://mlsanity.com/paper/2303.05082/) (2023)
- [Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training](https://mlsanity.com/paper/2305.18760/) (2023)
- [Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models](https://mlsanity.com/paper/2207.05928/) (2022)
- [Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding](https://mlsanity.com/paper/2311.06761/) (2023)
- [Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training](https://mlsanity.com/paper/2404.05560/) (2024)
- [SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models](https://mlsanity.com/paper/2604.12377/) (2026)
- [CNM-BERT: A Drop-In Structural Embedding for Chinese Characters via Ideographic Description Sequences](https://mlsanity.com/paper/2608.05167/) (2026)

## Abstract (arXiv)

Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese -- Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity manner. Specifically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We design a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multi-granularity representations. Our code will be available at https://github.com/alibaba/pretrained-language-models/LatticeBERT.