mlsanityData to 7 Oct 2026

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

arXiv 2112.10508 ↗First posted 20 Dec 2021cs.CL cs.LGcited by 130

Not on the map. Why?

Abstract (arXiv)

What are the units of text that we want to model? From bytes to multi-word expressions, text can be analyzed and generated at many granularities. Until recently, most natural language processing (NLP) models operated over words, treating those as discrete and atomic tokens, but starting with byte-pair encoding (BPE), subword-based approaches have become dominant in many areas, enabling small vocabularies while still allowing for fast inference. Is the end of the road character-level model or byte-level processing? In this survey, we connect several lines of work from the pre-neural and neural era, by showing how hybrid approaches of words and characters as well as subword-based approaches based on learned segmentation have been proposed and evaluated. We conclude that there is and likely will never be a silver bullet singular solution for all applications and that thinking seriously about tokenization remains important for many applications.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 106 of its 200 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

106 of 200 references.

References · 106

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 549
38 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  3. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  4. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  5. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  6. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  8. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  9. 2016Exploring the Limits of Language Modelingcited by 172
  10. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  11. 2015Character-Aware Neural Language Modelscited by 129
  12. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  13. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  14. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  15. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  16. 2020Fast WordPiece Tokenizationcited by 85
  17. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  18. 2014A Clockwork RNNcited by 70
  19. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  20. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  21. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  22. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  23. 2017Hash Embeddings for Efficient Word Representationscited by 54
  24. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  25. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  26. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  27. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  28. 2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
  29. 2019Glyce: Glyph-vectors for Chinese Character Representationscited by 46
  30. 2021Multi-view Subword Regularizationcited by 40
  31. 2020Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translationcited by 38
  32. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  33. 2016Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Losscited by 30
  34. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  35. 2019A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translationcited by 30
  36. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  37. 2021How BPE Affects Memorization in Transformerscited by 29
  38. 2015Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representationcited by 28

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 31; 2024: 32; 2025: 33; 2026: 15.

130 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  5. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  6. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  7. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  8. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  9. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  10. 2024Tokenization Is More Than Compressioncited by 87
  11. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
  12. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
38 more
  1. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 61
  2. 2025Large language models for automated scholarly paper review: A surveycited by 59
  3. 2023Tokenization and the Noiseless Channelcited by 58
  4. 2025SuperBPE: Space Travel for Language Modelscited by 58
  5. 2022Language Modelling with Pixelscited by 53
  6. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  7. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  8. 2022What do tokens know about their characters and how do they know it?cited by 36
  9. 2024How to Compute the Probability of a Wordcited by 36
  10. 2023Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logiccited by 34
  11. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  12. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  13. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  14. 2024Zero-Shot Tokenizer Transfercited by 30
  15. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  16. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  17. 2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
  18. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  19. 2023Text Rendering Strategies for Pixel Language Modelscited by 22
  20. 2022Text Generation with Text-Editing Modelscited by 21
  21. 2024Subobject-level Image Tokenizationcited by 20
  22. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  23. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  24. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  25. 2023Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologiescited by 16
  26. 2024Theoretical Analysis of Byte-Pair Encodingcited by 15
  27. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  28. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  29. 2025Tokenization is Sensitive to Language Variationcited by 12
  30. 2022UniMorph 4.0: Universal Morphologycited by 11
  31. 2022Incorporating Context into Subword Vocabulariescited by 11
  32. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  33. 2023What changes when you randomly choose BPE merge operations? Not muchcited by 10
  34. 2024Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Surveycited by 10
  35. 2023CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Codecited by 9
  36. 2024Did Translation Models Get More Robust Without Anyone Even Noticing?cited by 9
  37. 2023GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Explorationcited by 8
  38. 2024Quantum Transformer: Accelerating model inference via quantum linear algebracited by 8

50 most cited of 130 All 130 (JSON)