mlsanityData to 7 Oct 2026

Bridging the Gap for Tokenizer-Free Language Models

arXiv 1908.10322 ↗First posted 27 Aug 2019cs.CL cs.AI cs.IRcited by 21

Not on the map. Why?

Abstract (arXiv)

Purely character-based language models (LMs) have been lagging in quality on large scale datasets, and current state-of-the-art LMs rely on word tokenization. It has been assumed that injecting the prior knowledge of a tokenizer into the model is essential to achieving competitive results. In this paper, we show that contrary to this conventional wisdom, tokenizer-free LMs with sufficient capacity can achieve competitive performance on a large scale dataset. We train a vanilla transformer network with 40 self-attention layers on the One Billion Word (lm1b) benchmark and achieve a new state of the art for tokenizer-free LMs, pushing these models to be on par with their word-based counterparts.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 1; 2023: 5; 2024: 4; 2025: 3; 2026: 4.

21 papers.

Sort:

By area:

  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2023Challenges and Applications of Large Language Modelscited by 251
  3. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  4. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  5. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
  6. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  7. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  8. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  9. 2025Scaling Embedding Layers in Language Modelscited by 19
  10. 2024Training LLMs over Neurally Compressed Textcited by 16
  11. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  12. 2024Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Modelscited by 5
9 more
  1. 2025On the Origin of Algorithmic Progress in AIcited by 5
  2. 2023From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understandingcited by 4
  3. 2026Debugging code world modelscited by 4
  4. 2026Proxy Compression for Language Modelingcited by 3
  5. 2023Learning to Abstract with Nonparametric Variational Information Bottleneckcited by 2
  6. 2026Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansioncited by 2
  7. 2023Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritizationcited by 1
  8. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  9. 2022Small Character Models Match Large Word Models for Autocomplete Under Memory Constraintscited by 0