# Universal Word Segmentation: Implementation and Interpretation

arXiv: https://arxiv.org/abs/1807.02974 · HTML: https://mlsanity.com/paper/1807.02974/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2018.

## Built on by (most cited first, 10 of 10)

- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Neural Morphology Dataset and Models for Multiple Languages, from the Large to the Endangered](https://mlsanity.com/paper/2105.12428/) (2021)
- [Neural Token Segmentation for High Token-Internal Complexity](https://mlsanity.com/paper/2203.10845/) (2022)
- [Zero-shot hashtag segmentation for multilingual sentiment analysis](https://mlsanity.com/paper/2112.03213/) (2021)
- [Back to Patterns: Efficient Japanese Morphological Analysis with Feature-Sequence Trie](https://mlsanity.com/paper/2305.19045/) (2023)
- [Word Segmentation for Asian Languages: Chinese, Korean, and Japanese](https://mlsanity.com/paper/2407.19400/) (2024)
- [From LIMA to DeepLIMA: following a new path of interoperability](https://mlsanity.com/paper/2409.06550/) (2024)
- [Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text](https://mlsanity.com/paper/2608.18437/) (2026)
- [The Art of Hierarchical Competing Patterns: Gaussian Process Optimization of Hyphenation](https://mlsanity.com/paper/2609.07638/) (2026)

## Abstract (arXiv)

Word segmentation is a low-level NLP task that is non-trivial for a considerable number of languages. In this paper, we present a sequence tagging framework and apply it to word segmentation for a wide range of languages with different writing systems and typological characteristics. Additionally, we investigate the correlations between various typological factors and word segmentation accuracy. The experimental results indicate that segmentation accuracy is positively related to word boundary markers and negatively to the number of unique non-segmental terms. Based on the analysis, we design a small set of language-specific settings and extensively evaluate the segmentation system on the Universal Dependencies datasets. Our model obtains state-of-the-art accuracies on all the UD languages. It performs substantially better on languages that are non-trivial to segment, such as Chinese, Japanese, Arabic and Hebrew, when compared to previous work.