mlsanityData to 7 Oct 2026

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

arXiv 2404.00397 ↗First posted 30 Mar 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in machine translation implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to improve performance, and is even prone to incurring heavy degradation.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 17 of its 37 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

17 of 37 references.

References · 17

  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  7. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  8. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  9. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  10. 2016Efficient softmax approximation for GPUscited by 67
  11. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  12. 2023Tokenization and the Noiseless Channelcited by 58
5 more
  1. 2017Massive Exploration of Neural Machine Translation Architecturescited by 55
  2. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  3. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  4. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  5. 2017The University of Edinburgh's Neural MT Systems for WMT17cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 31
  2. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillationcited by 5
  3. 2026Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encodingcited by 1
  4. 2026From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attributioncited by 0
  5. 2026Workload-Driven Optimization for On-Device Real-Time Subtitle Translationcited by 0
  6. 2026Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Studycited by 0
  7. 2026Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruningcited by 0