mlsanityData to 7 Oct 2026

ByT5: Towards a token-free future with pre-trained byte-to-byte models

arXiv 2105.13626 ↗First posted 28 May 2021cs.CLcited by 504

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: they can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.

Key facts. Cited by 504 papers in the map (data through 7 Oct 2026). Builds on 55 of its 68 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

55 of 68 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
38 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2016HyperNetworkscited by 1,003
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  11. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  13. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  14. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  17. 2016Exploring the Limits of Language Modelingcited by 172
  18. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  19. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  20. 2015Character-Aware Neural Language Modelscited by 129
  21. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  22. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  23. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  24. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  25. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  26. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  27. 2016Neural Machine Translation in Linear Timecited by 65
  28. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  29. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  30. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  31. 2020SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflectioncited by 39
  32. 2016Recurrent Highway Networkscited by 36
  33. 2020Processing South Asian Languages Written in the Latin Script: the Dakshina Datasetcited by 35
  34. 2020Question Directed Graph Attention Network for Numerical Reasoning over Textcited by 35
  35. 2016Character-based Neural Machine Translationcited by 33
  36. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  37. 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
  38. 2015Multilingual Language Processing From Bytescited by 26

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 64; 2023: 76; 2024: 115; 2025: 111; 2026: 111.

504 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  11. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  12. 2022Confident Adaptive Language Modelingcited by 297
38 more
  1. 2022UL2: Unifying Language Learning Paradigmscited by 292
  2. 2023Challenges and Applications of Large Language Modelscited by 251
  3. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  4. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  5. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  6. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  7. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  8. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  9. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  10. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  11. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  12. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  13. 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 127
  14. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  15. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  16. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  17. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  18. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  19. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  20. 2021The Efficiency Misnomercited by 98
  21. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  22. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  23. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  24. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  25. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  26. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  27. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 67
  28. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  29. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  30. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  31. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  32. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  33. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 61
  34. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  35. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  36. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Modelscited by 58
  37. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  38. 2025SuperBPE: Space Travel for Language Modelscited by 58

50 most cited of 504 All 504 (JSON)