# Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

arXiv: https://arxiv.org/abs/2302.14220 · HTML: https://mlsanity.com/paper/2302.14220/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 30 of 33)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters](https://mlsanity.com/paper/2010.10392/) (2020)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation](https://mlsanity.com/paper/2104.08771/) (2021)
- [Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation](https://mlsanity.com/paper/2010.10907/) (2020)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://mlsanity.com/paper/1612.04629/) (2016)
- [Character-based Neural Machine Translation](https://mlsanity.com/paper/1603.00810/) (2016)
- [Revisiting Character-Based Neural Machine Translation with Capacity and Compression](https://mlsanity.com/paper/1808.09943/) (2018)
- [Saliency-driven Word Alignment Interpretation for Neural Machine Translation](https://mlsanity.com/paper/1906.10282/) (2019)
- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation](https://mlsanity.com/paper/1603.06147/) (2016)
- [Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT](https://mlsanity.com/paper/2109.01396/) (2021)

## Built on by (most cited first, 15 of 15)

- [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://mlsanity.com/paper/2404.14408/) (2024)
- [CUTE: Measuring LLMs' Understanding of Their Tokens](https://mlsanity.com/paper/2409.15452/) (2024)
- [Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?](https://mlsanity.com/paper/2409.19151/) (2024)
- [Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5](https://mlsanity.com/paper/2410.11627/) (2024)
- [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](https://mlsanity.com/paper/2310.01188/) (2023)
- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models](https://mlsanity.com/paper/2602.11174/) (2026)
- [CharED: Character-wise Ensemble Decoding for Large Language Models](https://mlsanity.com/paper/2407.11009/) (2024)
- [Improving Language and Modality Transfer in Translation by Character-level Modeling](https://mlsanity.com/paper/2505.24561/) (2025)
- [The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages](https://mlsanity.com/paper/2511.09197/) (2025)
- [MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages](https://mlsanity.com/paper/2601.16623/) (2026)
- [MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation](https://mlsanity.com/paper/2411.01474/) (2024)
- [Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs](https://mlsanity.com/paper/2605.01224/) (2026)
- [Neural Machine Translation for Low-Resource Tangkhul--English](https://mlsanity.com/paper/2606.25365/) (2026)
- [Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text](https://mlsanity.com/paper/2610.19294/) (2026)

## Abstract (arXiv)

Pretrained character-level and byte-level language models have been shown to be competitive with popular subword models across a range of Natural Language Processing (NLP) tasks. However, there has been little research on their effectiveness for neural machine translation (NMT), particularly within the popular pretrain-then-finetune paradigm. This work performs an extensive comparison across multiple languages and experimental conditions of character- and subword-level pretrained models (ByT5 and mT5, respectively) on NMT. We show the effectiveness of character-level modeling in translation, particularly in cases where fine-tuning data is limited. In our analysis, we show how character models' gains in translation quality are reflected in better translations of orthographically similar words and rare words. While evaluating the importance of source texts in driving model predictions, we highlight word-level patterns within ByT5, suggesting an ability to modulate word-level and character-level information during generation. We conclude by assessing the efficiency tradeoff of byte models, suggesting their usage in non-time-critical scenarios to boost translation quality.