# Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training

arXiv: https://arxiv.org/abs/2404.05560 · HTML: https://mlsanity.com/paper/2404.05560/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 59 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1810.07366/) (2018)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1904.09679/) (2019)
- [ChID: A Large-scale Chinese IDiom Dataset for Cloze Test](https://mlsanity.com/paper/1906.01265/) (2019)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter](https://mlsanity.com/paper/2105.07148/) (2021)
- [PERT: Pre-training BERT with Permuted Language Model](https://mlsanity.com/paper/2203.06906/) (2022)
- [AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization](https://mlsanity.com/paper/2008.11869/) (2020)
- [Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://mlsanity.com/paper/2104.07204/) (2021)
- [ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding](https://mlsanity.com/paper/2010.12148/) (2020)
- [A Novel Global Feature-Oriented Relational Triple Extraction Model based on Table Filling](https://mlsanity.com/paper/2109.06705/) (2021)
- [Neural Word Segmentation with Rich Pretraining](https://mlsanity.com/paper/1704.08960/) (2017)
- [Subword Encoding in Lattice LSTM for Chinese Word Segmentation](https://mlsanity.com/paper/1810.12594/) (2018)
- [Coupling Distant Annotation and Adversarial Training for Cross-Domain Chinese Word Segmentation](https://mlsanity.com/paper/2007.08186/) (2020)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)

## Abstract (arXiv)

Chinese sequence labeling tasks are heavily reliant on accurate word boundary demarcation. Although current pre-trained language models (PLMs) have achieved substantial gains on these tasks, they rarely explicitly incorporate boundary information into the modeling process. An exception to this is BABERT, which incorporates unsupervised statistical boundary information into Chinese BERT's pre-training objectives. Building upon this approach, we input supervised high-quality boundary information to enhance BABERT's learning, developing a semi-supervised boundary-aware PLM. To assess PLMs' ability to encode boundaries, we introduce a novel ``Boundary Information Metric'' that is both simple and effective. This metric allows comparison of different PLMs without task-specific fine-tuning. Experimental results on Chinese sequence labeling datasets demonstrate that the improved BABERT variant outperforms the vanilla version, not only on these tasks but also more broadly across a range of Chinese natural language understanding tasks. Additionally, our proposed metric offers a convenient and accurate means of evaluating PLMs' boundary awareness.