mlsanityData to 7 Oct 2026

Fast WordPiece Tokenization

arXiv 2012.15524 ↗First posted 31 Dec 2020cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

Tokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we propose efficient algorithms for the WordPiece tokenization used in BERT, from single-word tokenization to general text (e.g., sentence) tokenization. When tokenizing a single word, WordPiece uses a longest-match-first strategy, known as maximum matching. The best known algorithms so far are O(n^2) (where n is the input length) or O(nm) (where m is the maximum vocabulary token length). We propose a novel algorithm whose tokenization complexity is strictly O(n). Our method is inspired by the Aho-Corasick algorithm. We introduce additional linkages on top of the trie built from the vocabulary, allowing smart transitions when the trie matching cannot continue. For general text, we further propose an algorithm that combines pre-tokenization (splitting the text into words) and our linear-time WordPiece method into a single pass. Experimental results show that our method is 8.2x faster than HuggingFace Tokenizers and 5.1x faster than TensorFlow Text on average for general text tokenization.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 18; 2024: 25; 2025: 21; 2026: 12.

85 papers.

Sort:

By area:

  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  2. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  3. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  4. 2024LLMs are Good Sign Language Translatorscited by 76
  5. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  6. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  7. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  8. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  9. 2022A general-purpose material property data extraction pipeline from large polymer corpora using Natural Language Processingcited by 22
  10. 2024Subobject-level Image Tokenizationcited by 20
  11. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  12. 2022pNLP-Mixer: an Efficient all-MLP Architecture for Languagecited by 19
38 more
  1. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  2. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  3. 2023DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrewcited by 17
  4. 2025Multimodal Medical Code Tokenizercited by 14
  5. 2023ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Textscited by 13
  6. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  7. 2024POINTS1.5: Building a Vision-Language Model towards Real World Applicationscited by 12
  8. 2023What changes when you randomly choose BPE merge operations? Not muchcited by 10
  9. 2024Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languagescited by 10
  10. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
  11. 2022MaxMatch-Dropout: Subword Regularization for WordPiececited by 8
  12. 2023MultiLegalSBD: A Multilingual Legal Sentence Boundary Detection Datasetcited by 7
  13. 2025RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequencecited by 7
  14. 2025Serialized EHR make for good text representationscited by 7
  15. 2024An Interpretable Ensemble of Graph and Language Models for Improving Search Relevance in E-Commercecited by 6
  16. 2024Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokenscited by 6
  17. 2024Knowledge of Pretrained Language Models on Surface Information of Tokenscited by 5
  18. 2024ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Modelcited by 5
  19. 2024Tokenization as Finite-State Transductioncited by 5
  20. 2025Speculating LLMs' Chinese Training Data Pollution from Their Tokenscited by 5
  21. 2023Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabiccited by 4
  22. 2024A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Taskscited by 4
  23. 2024Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrappingcited by 4
  24. 2024Revisiting Character-level Adversarial Attacks for Language Modelscited by 4
  25. 2025Learning an Effective Premise Retrieval Model for Efficient Mathematical Formalizationcited by 4
  26. 2023Using Semantic Similarity and Text Embedding to Measure the Social Media Echo of Strategic Communicationscited by 3
  27. 2024I/O in Machine Learning Applications on HPC Systems: A 360-degree Surveycited by 3
  28. 2026Transducing Language Modelscited by 3
  29. 2026FreeTxt-Vi: A Benchmarked Vietnamese-English Toolkit for Segmentation, Sentiment, and Summarisationcited by 3
  30. 2022Language Model Classifier Aligns Better with Physician Word Sensitivity than XGBoost on Readmission Predictioncited by 2
  31. 2023BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognitioncited by 2
  32. 2023On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Modelcited by 2
  33. 2025Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiencycited by 2
  34. 2025MUTANT: A Recipe for Multilingual Tokenizer Designcited by 2
  35. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  36. 2021Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Modelscited by 1
  37. 2023Multiple Relations Classification using Imbalanced Predictions Adaptationcited by 1
  38. 2023Advancing Perception in Artificial Intelligence through Principles of Cognitive Sciencecited by 1

50 most cited of 85 All 85 (JSON)