mlsanityData to 14 Oct 2026

Which Pieces Does Unigram Tokenization Really Need?

arXiv 2512.12641 ↗First posted 14 Dec 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof. We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices. We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 8 of its 10 references that are in the map. First posted on arXiv 14 Dec 2025.

Builds on

8 of 10 references.

References · 8

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  4. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  5. 2024Tokenization Is More Than Compressioncited by 89
  6. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 31
  7. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  8. 2025BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenizationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  2. 2026What Language is This? Ask Your Tokenizercited by 2
  3. 2026LangMAP: A Language-Adaptive Approach to Tokenizationcited by 0
  4. 2026Explicit Boundary Markers for Subword Vocabulariescited by 0