mlsanityData to 7 Oct 2026

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT

arXiv 2306.01393 ↗First posted 2 Jun 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Subword tokenization is the de facto standard for tokenization in neural language models and machine translation systems. Three advantages are frequently cited in favor of subwords: shorter encoding of frequent tokens, compositionality of subwords, and ability to deal with unknown words. As their relative importance is not entirely clear yet, we propose a tokenization approach that enables us to separate frequency (the first advantage) from compositionality. The approach uses Huffman coding to tokenize words, by order of frequency, using a fixed amount of symbols. Experiments with CS-DE, EN-FR and EN-DE NMT show that frequency alone accounts for 90%-95% of the scores reached by BPE, hence compositionality has less importance than previously thought.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 57 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

20 of 57 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  9. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  10. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  11. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  12. 2014Addressing the Rare Word Problem in Neural Machine Translationcited by 50
8 more
  1. 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
  2. 2021Why don't people use character-level machine translation?cited by 27
  3. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  4. 2016Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Modelscited by 18
  5. 2017Stronger Baselines for Trustable Results in Neural Machine Translationcited by 12
  6. 2021Subword Mapping and Anchoring across Languagescited by 12
  7. 2019Character-based NMT with Transformercited by 10
  8. 2020On Romanization for Model Transfer Between Scripts in Neural Machine Translationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  2. 2024MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compressioncited by 1
  3. 2024Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenizationcited by 0
  4. 2026Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluationcited by 0