# Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling

arXiv: https://arxiv.org/abs/1810.03167 · HTML: https://mlsanity.com/paper/1810.03167/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2018.

## Built on by (most cited first, 12 of 12)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [A Masked Segmental Language Model for Unsupervised Natural Language Segmentation](https://mlsanity.com/paper/2104.07829/) (2021)
- [Subword Segmental Language Modelling for Nguni Languages](https://mlsanity.com/paper/2210.06525/) (2022)
- [Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation](https://mlsanity.com/paper/2305.07005/) (2023)
- [Unsupervised Word Segmentation with Bi-directional Neural Language Model](https://mlsanity.com/paper/2103.01421/) (2021)
- [Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation](https://mlsanity.com/paper/2403.07567/) (2024)
- [Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention](https://mlsanity.com/paper/2102.01223/) (2021)
- [The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages](https://mlsanity.com/paper/2511.09197/) (2025)
- [Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages](https://mlsanity.com/paper/2110.08415/) (2021)
- [SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translation](https://mlsanity.com/paper/2307.16400/) (2023)
- [Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models](https://mlsanity.com/paper/2505.19631/) (2025)
- [Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining](https://mlsanity.com/paper/2609.01151/) (2026)

## Abstract (arXiv)

Previous traditional approaches to unsupervised Chinese word segmentation (CWS) can be roughly classified into discriminative and generative models. The former uses the carefully designed goodness measures for candidate segmentation, while the latter focuses on finding the optimal segmentation of the highest generative probability. However, while there exists a trivial way to extend the discriminative models into neural version by using neural language models, those of generative ones are non-trivial. In this paper, we propose the segmental language models (SLMs) for CWS. Our approach explicitly focuses on the segmental nature of Chinese, as well as preserves several properties of language models. In SLMs, a context encoder encodes the previous context and a segment decoder generates each segment incrementally. As far as we know, we are the first to propose a neural model for unsupervised CWS and achieve competitive performance to the state-of-the-art statistical models on four different datasets from SIGHAN 2005 bakeoff.