# MaxMatch-Dropout: Subword Regularization for WordPiece

arXiv: https://arxiv.org/abs/2209.04126 · HTML: https://mlsanity.com/paper/2209.04126/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 9 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Fast WordPiece Tokenization](https://mlsanity.com/paper/2012.15524/) (2020)
- [Evaluating Robustness to Input Perturbations for Neural Machine Translation](https://mlsanity.com/paper/2005.00580/) (2020)
- [Joint Optimization of Tokenization and Downstream Model](https://mlsanity.com/paper/2105.12410/) (2021)
- [Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation](https://mlsanity.com/paper/2203.13528/) (2022)

## Built on by (most cited first, 8 of 8)

- [Incorporating Context into Subword Vocabularies](https://mlsanity.com/paper/2210.07095/) (2022)
- [StochasTok: Improving Fine-Grained Subword Understanding in LLMs](https://mlsanity.com/paper/2506.01687/) (2025)
- [We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text](https://mlsanity.com/paper/2404.07304/) (2024)
- [Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data](https://mlsanity.com/paper/2512.23422/) (2025)
- [Distributional Properties of Subword Regularization](https://mlsanity.com/paper/2408.11443/) (2024)
- [SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization](https://mlsanity.com/paper/2409.06216/) (2024)
- [Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing](https://mlsanity.com/paper/2304.10808/) (2023)
- [Tokenization Preference for Human and Machine Learning Model: An Annotation Study](https://mlsanity.com/paper/2304.10813/) (2023)

## Abstract (arXiv)

We present a subword regularization method for WordPiece, which uses a maximum matching algorithm for tokenization. The proposed method, MaxMatch-Dropout, randomly drops words in a search using the maximum matching algorithm. It realizes finetuning with subword regularization for popular pretrained language models such as BERT-base. The experimental results demonstrate that MaxMatch-Dropout improves the performance of text classification and machine translation tasks as well as other subword regularization methods. Moreover, we provide a comparative analysis of subword regularization methods: subword regularization with SentencePiece (Unigram), BPE-Dropout, and MaxMatch-Dropout.