# Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text

arXiv: https://arxiv.org/abs/2608.18437 · HTML: https://mlsanity.com/paper/2608.18437/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 40 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 6 of 6)

- [TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer](https://mlsanity.com/paper/2210.11753/) (2022)
- [Universal Word Segmentation: Implementation and Interpretation](https://mlsanity.com/paper/1807.02974/) (2018)
- [State-of-the-art Chinese Word Segmentation with Bi-LSTMs](https://mlsanity.com/paper/1808.06511/) (2018)
- [Deciphering Undersegmented Ancient Scripts Using Phonetic Prior](https://mlsanity.com/paper/2010.11054/) (2020)
- [Neural Word Segmentation Learning for Chinese](https://mlsanity.com/paper/1606.04300/) (2016)
- [Convolutional Neural Network with Word Embeddings for Chinese Word Segmentation](https://mlsanity.com/paper/1711.04411/) (2017)

## Abstract (arXiv)

Tangut is an extinct language whose script does not explicitly mark word boundaries. We present the first systematic study of Tangut word segmentation using 2,750 expert-annotated segments (31,893 tokens), traditional lexicons, and unlabeled text. Our framework combines a reliability-calibrated lexicon-lattice representation, explicit distributional statistics, and a lightweight character encoder pretrained with MLM. In within-source five-fold cross-validation, the model integrating TangutEncoder, CRF, and external features obtains the numerically highest main-system mean F1 of 0.911 and substantially improves recall beyond the labeled training vocabulary. We further evaluate document-level transfer on 479 segments (4081 tokens) from five works absent from the annotated training corpus. You can access our project at https://github.com/jiangli-va/TangutSeg.