mlsanityData to 7 Oct 2026

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

arXiv 2409.04599 ↗First posted 6 Sep 2024cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that our method does not reduce the downstream performance, and in several cases improves it.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

28 of 50 references.

References · 28

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  10. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  11. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  12. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
16 more
  1. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  2. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  3. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  4. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  5. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  6. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  7. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  8. 2023Tokenization and the Noiseless Channelcited by 58
  9. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  10. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  11. 2024Toward a Theory of Tokenization in LLMscited by 28
  12. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
  13. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 19
  14. 2017The University of Edinburgh's Neural MT Systems for WMT17cited by 16
  15. 2024Flexibly Scaling Large Language Models Contexts Through Extensible Tokenizationcited by 6
  16. 2024Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 17.

30 papers.

Sort:

By area:

  1. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 31
  2. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  3. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  4. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  5. 2025A Partition Cover Approach to Tokenizationcited by 5
  6. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  7. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  8. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  9. 2025Which Pieces Does Unigram Tokenization Really Need?cited by 4
  10. 2026Tokenisation via Convex Relaxationscited by 4
  11. 2026AdaptBPE: From General Purpose to Specialized Tokenizerscited by 3
  12. 2025Incorporating Domain Knowledge into Materials Tokenizationcited by 2
18 more
  1. 2025Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insightscited by 2
  2. 2025Lossless Vocabulary Reduction for Auto-Regressive Language Modelscited by 2
  3. 2025MUTANT: A Recipe for Multilingual Tokenizer Designcited by 2
  4. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  5. 2025Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMscited by 2
  6. 2026Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokenscited by 2
  7. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  8. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  9. 2026From Lemmas to Dependencies: What Signals Drive Light Verbs Classification?cited by 1
  10. 2026Faster Superword Tokenizationcited by 1
  11. 2026From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attributioncited by 0
  12. 2026Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translationcited by 0
  13. 2026BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysiscited by 0
  14. 2026What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?cited by 0
  15. 2026When Tokenization is Secretly Output Supervisioncited by 0
  16. 2026Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruningcited by 0
  17. 2026Objective vs. Search: Decomposing What Makes a Good Tokenisercited by 0
  18. 2026Counting and Min-Cost Encoding for Tokenization in Large Language Modelscited by 0