# Why don't people use character-level machine translation?

arXiv: https://arxiv.org/abs/2110.08191 · HTML: https://mlsanity.com/paper/2110.08191/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 74 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Tagged Back-Translation](https://mlsanity.com/paper/1906.06442/) (2019)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation](https://mlsanity.com/paper/2105.08504/) (2021)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://mlsanity.com/paper/1612.04629/) (2016)
- [Training Deeper Neural Machine Translation Models with Transparent Attention](https://mlsanity.com/paper/1808.07561/) (2018)
- [Revisiting Character-Based Neural Machine Translation with Capacity and Compression](https://mlsanity.com/paper/1808.09943/) (2018)
- [How Decoding Strategies Affect the Verifiability of Generated Text](https://mlsanity.com/paper/1911.03587/) (2019)
- [A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation](https://mlsanity.com/paper/1603.06147/) (2016)

## Built on by (most cited first, 27 of 27)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [Zero-Shot Tokenizer Transfer](https://mlsanity.com/paper/2405.07883/) (2024)
- [CUTE: Measuring LLMs' Understanding of Their Tokens](https://mlsanity.com/paper/2409.15452/) (2024)
- [ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models](https://mlsanity.com/paper/2212.10474/) (2022)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation](https://mlsanity.com/paper/2302.14220/) (2023)
- [Tokenization is Sensitive to Language Variation](https://mlsanity.com/paper/2502.15343/) (2025)
- [Effects of sub-word segmentation on performance of transformer language models](https://mlsanity.com/paper/2305.05480/) (2023)
- [Large Vocabulary Size Improves Large Language Models](https://mlsanity.com/paper/2406.16508/) (2024)
- [Subword-Delimited Downsampling for Better Character-Level Translation](https://mlsanity.com/paper/2212.01304/) (2022)
- [Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation](https://mlsanity.com/paper/2305.07005/) (2023)
- [Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora](https://mlsanity.com/paper/2305.17906/) (2023)
- [Exploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation](https://mlsanity.com/paper/2309.12491/) (2023)
- [Overcoming Vocabulary Constraints with Pixel-level Fallback](https://mlsanity.com/paper/2504.02122/) (2025)
- [Local Byte Fusion for Neural Machine Translation](https://mlsanity.com/paper/2205.11490/) (2022)
- [Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT](https://mlsanity.com/paper/2306.01393/) (2023)
- [UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8](https://mlsanity.com/paper/2511.05578/) (2025)
- [Improving Language and Modality Transfer in Translation by Character-level Modeling](https://mlsanity.com/paper/2505.24561/) (2025)
- [Syllabic Agglutinative Tokenizations for Indonesian LLM: A Study from Gasing Literacy Learning System](https://mlsanity.com/paper/2601.11643/) (2026)
- [CharSpan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages](https://mlsanity.com/paper/2305.05214/) (2023)
- [Making Sentence Embeddings Robust to User-Generated Content](https://mlsanity.com/paper/2403.17220/) (2024)
- [Character-level NMT and language similarity](https://mlsanity.com/paper/2308.04398/) (2023)
- [The Query Channel: Information-Theoretic Limits of Masking-Based Explanations](https://mlsanity.com/paper/2604.16689/) (2026)
- [Toppling the Hierarchy in Byte-level Language Modeling](https://mlsanity.com/paper/2609.00463/) (2026)

## Abstract (arXiv)

We present a literature and empirical survey that critically assesses the state of the art in character-level modeling for machine translation (MT). Despite evidence in the literature that character-level systems are comparable with subword systems, they are virtually never used in competitive setups in WMT competitions. We empirically show that even with recent modeling innovations in character-level natural language processing, character-level MT systems still struggle to match their subword-based counterparts. Character-level MT systems show neither better domain robustness, nor better morphological generalization, despite being often so motivated. However, we are able to show robustness towards source side noise and that translation quality does not degrade with increasing beam size at decoding time.