# Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models

arXiv: https://arxiv.org/abs/2505.19631 · HTML: https://mlsanity.com/paper/2505.19631/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 52 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 16 of 16)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2305.16582/) (2023)
- [Language Model is All You Need: Natural Language Understanding as Question Answering](https://mlsanity.com/paper/2011.03023/) (2020)
- [Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2312.08762/) (2023)
- [Sequence Modeling via Segmentations](https://mlsanity.com/paper/1702.07463/) (2017)
- [Seeking Common but Distinguishing Difference, A Joint Aspect-based Sentiment Analysis Model](https://mlsanity.com/paper/2111.09634/) (2021)
- [Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling](https://mlsanity.com/paper/1810.03167/) (2018)
- [Explicit Sentence Compression for Neural Machine Translation](https://mlsanity.com/paper/1912.11980/) (2019)
- [Neural Word Segmentation Learning for Chinese](https://mlsanity.com/paper/1606.04300/) (2016)
- [Unsupervised Word Segmentation with Bi-directional Neural Language Model](https://mlsanity.com/paper/2103.01421/) (2021)
- [Bi-directional LSTM Recurrent Neural Network for Chinese Word Segmentation](https://mlsanity.com/paper/1602.04874/) (2016)
- [Convolutional Neural Network with Word Embeddings for Chinese Word Segmentation](https://mlsanity.com/paper/1711.04411/) (2017)
- [Effective Subword Segmentation for Text Comprehension](https://mlsanity.com/paper/1811.02364/) (2018)
- [Language Modelling Approaches to Adaptive Machine Translation](https://mlsanity.com/paper/2401.14559/) (2024)
- [Chinese Word Segmentation: Another Decade Review (2007-2017)](https://mlsanity.com/paper/1901.06079/) (2019)

## Built on by (most cited first, 1 of 1)

- [ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding](https://mlsanity.com/paper/2609.17241/) (2026)

## Abstract (arXiv)

Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA (Large Language Model-Inspired Aho-Corasick Automaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic n-gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA