mlsanityData to 14 Oct 2026

Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?

arXiv 2310.17774 ↗First posted 26 Oct 2023cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

An important assumption that comes with using LLMs on psycholinguistic data has gone unverified. LLM-based predictions are based on subword tokenization, not decomposition of words into morphemes. Does that matter? We carefully test this by comparing surprisal estimates using orthographic, morphological, and BPE tokenization against reading time data. Our results replicate previous findings and provide evidence that in the aggregate, predictions using BPE tokenization do not suffer relative to morphological and orthographic segmentation. However, a finer-grained analysis points to potential issues with relying on BPE-based tokenization, as well as providing promising results involving morphologically-aware surprisal estimates and suggesting a new method for evaluating morphological prediction.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 9 of its 48 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

9 of 48 references.

References · 9

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  3. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  4. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  5. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  6. 2017The Natural Stories Corpuscited by 40
  7. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 27
  8. 2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
  9. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  2. 2025Large Language Models Are Human-Like Internallycited by 29
  3. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  4. 2023Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?cited by 7
  5. 2024Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically alignedcited by 5
  6. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  7. 2026Transducing Language Modelscited by 5
  8. 2026Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisalcited by 1
  9. 2025How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usagescited by 0
  10. 2026On the scaling relationship between cloze probabilities and language model next-token predictioncited by 0
  11. 2026Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?cited by 0
  12. 2026surprisal is Not a Theorycited by 0
1 more
  1. 2026From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanishcited by 0