mlsanityData to 7 Oct 2026

From Language Models over Tokens to Language Models over Characters

arXiv 2412.03719 ↗First posted 4 Dec 2024cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Modern language models are internally -- and mathematically -- distributions over ittoken strings rather than itcharacter strings, posing numerous challenges for programmers building user applications on top of them. For example, if a prompt is specified as a character string, it must be tokenized before passing it to the token-level language model. Thus, the tokenizer and consequent processing are very sensitive to the specification of the prompt (e.g., whether the prompt ends with a space or not). This paper presents algorithms for converting token-level language models to character-level ones. We present both exact and approximate algorithms. In the empirical portion of the paper, we benchmark the practical runtime and approximation quality. Across four publicly available language models, we find that -- even with a small computation budget -- our method is able to accurately approximate the character-level distribution at reasonably fast speeds, and that a significant improvement in the language model's compression rate (bits/byte) is achieved.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

20 of 37 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  9. 2023Efficient Guided Generation for Large Language Modelscited by 192
  10. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  11. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  12. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
8 more
  1. 2024Automata-based constraints for language model decodingcited by 49
  2. 2024How to Compute the Probability of a Wordcited by 36
  3. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  4. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  5. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  6. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  7. 2023Should you marginalize over possible tokenizations?cited by 17
  8. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 10.

28 papers.

Sort:

By area:

  1. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  2. 2025SuperBPE: Space Travel for Language Modelscited by 58
  3. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  4. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  5. 2025Causal Estimation of Tokenisation Biascited by 16
  6. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  7. 2025Self-Steering Language Modelscited by 13
  8. 2025Adversarial Tokenizationcited by 11
  9. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  10. 2025The Harmonic Structure of Information Contourscited by 7
  11. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  12. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
16 more
  1. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  2. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillationcited by 5
  3. 2025Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capabilitycited by 3
  4. 2026Proxy Compression for Language Modelingcited by 3
  5. 2026Transducing Language Modelscited by 3
  6. 2025A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behaviorcited by 2
  7. 2025Lossless Vocabulary Reduction for Auto-Regressive Language Modelscited by 2
  8. 2026Ensembling Language Models with Sequential Monte Carlocited by 2
  9. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2
  10. 2025Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Modelscited by 1
  11. 2025Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-servicecited by 0
  12. 2026Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspectivecited by 0
  13. 2026Probing for Reading Timescited by 0
  14. 2026surprisal is Not a Theorycited by 0
  15. 2026Language Models are not Equally Robust to Non-Canonical Tokenization across Languagescited by 0
  16. 2026Stochastic Estimation of Transduced Language Modelscited by 0