mlsanityData to 7 Oct 2026

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

arXiv 2302.14220 ↗First posted 28 Feb 2023cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Pretrained character-level and byte-level language models have been shown to be competitive with popular subword models across a range of Natural Language Processing (NLP) tasks. However, there has been little research on their effectiveness for neural machine translation (NMT), particularly within the popular pretrain-then-finetune paradigm. This work performs an extensive comparison across multiple languages and experimental conditions of character- and subword-level pretrained models (ByT5 and mT5, respectively) on NMT. We show the effectiveness of character-level modeling in translation, particularly in cases where fine-tuning data is limited. In our analysis, we show how character models' gains in translation quality are reflected in better translations of orthographically similar words and rare words. While evaluating the importance of source texts in driving model predictions, we highlight word-level patterns within ByT5, suggesting an ability to modulate word-level and character-level information during generation. We conclude by assessing the efficiency tradeoff of byte models, suggesting their usage in non-time-critical scenarios to boost translation quality.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

33 of 54 references.

References · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  10. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  11. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  12. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
21 more
  1. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  2. 2016Exploring the Limits of Language Modelingcited by 172
  3. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  4. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  5. 2015Character-Aware Neural Language Modelscited by 129
  6. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  7. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  8. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  9. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  10. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  11. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  12. 2016How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairscited by 39
  13. 2016Character-based Neural Machine Translationcited by 33
  14. 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
  15. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  16. 2021Why don't people use character-level machine translation?cited by 27
  17. 2016A Character-Level Decoder without Explicit Segmentation for Neural Machine Translationcited by 21
  18. 2021Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMTcited by 21
  19. 2019Towards Understanding Neural Machine Translation with Word Importancecited by 19
  20. 2022Subword-Delimited Downsampling for Better Character-Level Translationcited by 7
  21. 2021Analyzing the Use of Character-Level Translation with Sparse and Noisy Datasetscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 2; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  2. 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 28
  3. 2024Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?cited by 23
  4. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  5. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 6
  6. 2024Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representationscited by 5
  7. 2024CharED: Character-wise Ensemble Decoding for Large Language Modelscited by 3
  8. 2026The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Modelscited by 3
  9. 2025The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languagescited by 2
  10. 2025Improving Language and Modality Transfer in Translation by Character-level Modelingcited by 1
  11. 2026MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languagescited by 1
  12. 2024MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translationcited by 0
2 more
  1. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
  2. 2026Neural Machine Translation for Low-Resource Tangkhul--Englishcited by 0