mlsanityData to 7 Oct 2026

How to Compute the Probability of a Word

arXiv 2406.14561 ↗First posted 20 Jun 2024cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) estimate a probability distribution over strings in a natural language; these distributions are crucial for computing perplexity and surprisal in linguistics research. While we are usually concerned with measuring these values for words, most LMs operate over subwords. Despite seemingly straightforward, accurately computing probabilities over one unit given probabilities over the other requires care. Indeed, we show here that many recent linguistic studies have been incorrectly computing these values. This paper derives the correct methods for computing word probabilities, highlighting issues when relying on language models that use beginning-of-word (bow)-marking tokenisers, e.g., the GPT family. Empirically, we show that correcting the widespread bug in probability computations affects measured outcomes in sentence comprehension and lexical optimisation analyses.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 17 of its 51 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

17 of 51 references.

References · 17

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  6. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  7. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  8. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  9. 2023Tokenization and the Noiseless Channelcited by 58
  10. 2017The Natural Stories Corpuscited by 40
  11. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
  12. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 32
5 more
  1. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  2. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  3. 2022On the Effect of Anticipation on Reading Timescited by 18
  4. 2023Should you marginalize over possible tokenizations?cited by 17
  5. 2023Revisiting the Optimality of Word Lengthscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 14; 2026: 13.

36 papers.

Sort:

By area:

  1. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  2. 2024From Language Models over Tokens to Language Models over Characterscited by 28
  3. 2025Causal Estimation of Tokenisation Biascited by 16
  4. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  5. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  6. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  7. 2024Tokenisation is NP-Completecited by 10
  8. 2024Towards a Similarity-adjusted Surprisal Theorycited by 9
  9. 2025Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Modelscited by 9
  10. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  11. 2025Subword models struggle with word learning, but surprisal hides itcited by 6
  12. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
24 more
  1. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillationcited by 5
  2. 2025Probing Syntax in Large Language Models: Successes and Remaining Challengescited by 4
  3. 2025The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakagecited by 3
  4. 2025How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?cited by 3
  5. 2025The Unheard Alternative: Contrastive Explanations for Speech-to-Text Modelscited by 3
  6. 2025Convergence and Divergence of Language Models under Different Random Seedscited by 3
  7. 2026Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?cited by 3
  8. 2026Transducing Language Modelscited by 3
  9. 2024A Grounded Typology of Word Classescited by 2
  10. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2
  11. 2024Reverse-Engineering the Readercited by 1
  12. 2025Lost in Space: Finding the Right Tokens for Structured Outputcited by 1
  13. 2025Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorlycited by 1
  14. 2026Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasetscited by 1
  15. 2026Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisalcited by 1
  16. 2026An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisalcited by 1
  17. 2025Do Generalisation Results Generalise?cited by 0
  18. 2026On the scaling relationship between cloze probabilities and language model next-token predictioncited by 0
  19. 2026Traces of Social Competence in Large Language Modelscited by 0
  20. 2026To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Timescited by 0
  21. 2026Probing for Reading Timescited by 0
  22. 2026The Frequency Confound in Language-Model Surprisal and Metaphor Noveltycited by 0
  23. 2026Beetle: A Bilingual Model Suite for Modelling Second-Language Processingcited by 0
  24. 2026Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluationcited by 0