# Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling

arXiv: https://arxiv.org/abs/2210.15231 · HTML: https://mlsanity.com/paper/2210.15231/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [Glyce: Glyph-vectors for Chinese Character Representations](https://mlsanity.com/paper/1901.10125/) (2019)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter](https://mlsanity.com/paper/2105.07148/) (2021)
- [AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization](https://mlsanity.com/paper/2008.11869/) (2020)
- [ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding](https://mlsanity.com/paper/2010.12148/) (2020)
- [Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF](https://mlsanity.com/paper/1704.01314/) (2017)
- [Subword Encoding in Lattice LSTM for Chinese Word Segmentation](https://mlsanity.com/paper/1810.12594/) (2018)
- [Coupling Distant Annotation and Adversarial Training for Cross-Domain Chinese Word Segmentation](https://mlsanity.com/paper/2007.08186/) (2020)
- [Combining Discrete and Neural Features for Sequence Labeling](https://mlsanity.com/paper/1708.07279/) (2017)

## Built on by (most cited first, 2 of 2)

- [Entropy-Driven Pre-Tokenization for Byte-Pair Encoding](https://mlsanity.com/paper/2506.15889/) (2025)
- [Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training](https://mlsanity.com/paper/2404.05560/) (2024)

## Abstract (arXiv)

Boundary information is critical for various Chinese language processing tasks, such as word segmentation, part-of-speech tagging, and named entity recognition. Previous studies usually resorted to the use of a high-quality external lexicon, where lexicon items can offer explicit boundary information. However, to ensure the quality of the lexicon, great human effort is always necessary, which has been generally ignored. In this work, we suggest unsupervised statistical boundary information instead, and propose an architecture to encode the information directly into pre-trained language models, resulting in Boundary-Aware BERT (BABERT). We apply BABERT for feature induction of Chinese sequence labeling tasks. Experimental results on ten benchmarks of Chinese sequence labeling demonstrate that BABERT can provide consistent improvements on all datasets. In addition, our method can complement previous supervised lexicon exploration, where further improvements can be achieved when integrated with external lexicon information.