# Unsupervised Tokenization Learning

arXiv: https://arxiv.org/abs/2205.11443 · HTML: https://mlsanity.com/paper/2205.11443/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 39 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Unsupervised Learning of Temporal Abstractions with Slot-based Transformers](https://mlsanity.com/paper/2203.13573/) (2022)
- [Learning Language from a Large (Unannotated) Corpus](https://mlsanity.com/paper/1401.3372/) (2014)
- [Brain Principles Programming](https://mlsanity.com/paper/2202.12710/) (2022)
- [Constructing Financial Sentimental Factors in Chinese Market Using Natural Language Processing](https://mlsanity.com/paper/1809.08390/) (2018)
- [Natural Language Generation Using Link Grammar for General Conversational Intelligence](https://mlsanity.com/paper/2105.00830/) (2021)

## Built on by (most cited first, 2 of 2)

- [Self-tuning hyper-parameters for unsupervised cross-lingual tokenization](https://mlsanity.com/paper/2303.02427/) (2023)
- [Evolution of Efficient Symbolic Communication Codes](https://mlsanity.com/paper/2306.02383/) (2023)

## Abstract (arXiv)

In the presented study, we discover that the so-called "transition freedom" metric appears superior for unsupervised tokenization purposes in comparison to statistical metrics such as mutual information and conditional probability, providing F-measure scores in range from 0.71 to 1.0 across explored multilingual corpora. We find that different languages require different offshoots of that metric (such as derivative, variance, and "peak values") for successful tokenization. Larger training corpora do not necessarily result in better tokenization quality, while compressing the models by eliminating statistically weak evidence tends to improve performance. The proposed unsupervised tokenization technique provides quality better than or comparable to lexicon-based ones, depending on the language.