mlsanityData to 14 Oct 2026

A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese

arXiv 2610.04898 ↗First posted 4 Oct 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This study analyzes the predictive power of LM-derived, token-level surprisal on Mandarin Chinese reading times. We first propose the Shortest Matching Sequence (SMS), an alignment scheme that maps between the word segmentation assumed by eye-tracking corpora and the LMs' subword tokenization, as the two tokenizations often disagree in the context of Mandarin Chinese. Then, using a suite of Chinese-Pythia models (14M-1.4B) trained on scratch with 30B tokens, we examine how well surprisal predicts first fixation duration, gaze duration, and total reading time in three paragraph-level eye-tracking corpora of Mandarin Chinese (GECO-CN, HKP, and MECO). Contrary to previous null findings, our results show that surprisal is predictive of Chinese reading times. However, whether predictive power scales with model size and the amount of training is corpus-specific: bigger models predict better in GECO-CN, whereas inverse scaling emerges in HKP and, at the largest sizes, in MECO. Subsequently, we tested one possible explanation for the inverse scaling in HKP and found that checkpoints whose surprisal remains closer to n-gram statistics are better predictors of reading. All in all, the predictive power of surprisal on Chinese reading time measurements is corpus-specific, which cautions against drawing scaling conclusions from a single corpus.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 46 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

16 of 46 references.

References · 16

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  4. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  5. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  6. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  7. 2021Revisiting the Uniform Information Density Hypothesiscited by 68
  8. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  9. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  10. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  11. 2025Large Language Models Are Human-Like Internallycited by 29
  12. 2024On the Role of Context in Reading Time Predictioncited by 20
4 more
  1. 2022On the Effect of Anticipation on Reading Timescited by 19
  2. 2025The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakagecited by 5
  3. 2024Attention-aware semantic relevance predicting Chinese sentence readingcited by 4
  4. 2024Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differencescited by 2

Built on by

Not cited yet.