Data to 7 Oct 2026

BPE-Dropout: Simple and Effective Subword Regularization

arXiv 1910.13267 ↗First posted 29 Oct 2019cs.CLcited by 189

Not on the map. Why?

Abstract (arXiv)

Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create another subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself incorporates the ability to produce multiple segmentations of the same word. We introduce BPE-dropout - simple and effective subword regularization method based on and compatible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmentations within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves translation quality up to 3 BLEU compared to BPE and up to 0.9 BLEU compared to the previous subword regularization.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 29; 2023: 32; 2024: 42; 2025: 28; 2026: 30.

189 papers.

Sort:

By area:

2021Zero-Shot Text-to-Image Generationcited by 4,276
2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
2022Efficient Training of Language Models to Fill in the Middlecited by 248
2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
2023Language Modeling Is Compressioncited by 225
2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158

38 more

2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
2021Survey of Low-Resource Machine Translationcited by 92
2024Tokenization Is More Than Compressioncited by 87
2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
2023Tokenization and the Noiseless Channelcited by 58
2025SuperBPE: Space Travel for Language Modelscited by 58
2022Language Modelling with Pixelscited by 53
2021HerBERT: Efficiently Pretrained Transformer-based Language Model for Polishcited by 51
2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
2021Multi-view Subword Regularizationcited by 40
2021Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Modelscited by 39
2022What do tokens know about their characters and how do they know it?cited by 36
2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Informationcited by 32
2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
2021How BPE Affects Memorization in Transformerscited by 29
2024From Language Models over Tokens to Language Models over Characterscited by 28
2021Why don't people use character-level machine translation?cited by 27
2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
2024h4rm3l: A language for Composable Jailbreak Attack Synthesiscited by 23
2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 23
2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
2023UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human languagecited by 22

50 most cited of 189 All 189 (JSON)