mlsanityData to 14 Oct 2026

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

arXiv 2505.24689 ↗First posted 30 May 2025cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Byte Pair Encoding (BPE) tokenizers, widely used in Large Language Models, face challenges in multilingual settings, including penalization of non-Western scripts and the creation of tokens with partial UTF-8 sequences. Pretokenization, often reliant on complex regular expressions, can also introduce fragility and unexpected edge cases. We propose SCRIPT (Script Category Representation in PreTokenization), a novel encoding scheme that bypasses UTF-8 byte conversion by using initial tokens based on Unicode script and category properties. This approach enables a simple, rule-based pretokenization strategy that respects script boundaries, offering a robust alternative to pretokenization strategies based on regular expressions. We also introduce and validate a constrained BPE merging strategy that enforces character integrity, applicable to both SCRIPT-BPE and byte-based BPE. Our experiments demonstrate that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties for non-Latin-script languages.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

14 of 19 references.

References · 14

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  4. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  5. 2024Tokenization Is More Than Compressioncited by 89
  6. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  7. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  8. 2025SuperBPE: Space Travel for Language Modelscited by 63
  9. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  10. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 32
  11. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  12. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 25
2 more
  1. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  2. 2025Tokenization is Sensitive to Language Variationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  3. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  4. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  5. 2025Which Pieces Does Unigram Tokenization Really Need?cited by 4
  6. 2026Tokenization with Split Treescited by 2
  7. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  8. 2026Faster Superword Tokenizationcited by 1
  9. 2026Beyond Perplexity: UTF-8 Validity in Byte-aware Language Modelscited by 0
  10. 2026Explicit Boundary Markers for Subword Vocabulariescited by 0
  11. 2026The Functionalizer: Lossless Functional Decomposition for Subword Tokenizationcited by 0
  12. 2026Are Language Models Script-Aware?cited by 0