mlsanityData to 7 Oct 2026

Lower Perplexity is Not Always Human-Like

arXiv 2106.01229 ↗First posted 2 Jun 2021cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

In computational psycholinguistics, various language models have been evaluated against human reading behavior (e.g., eye movement) to build human-like computational models. However, most previous efforts have focused almost exclusively on English, despite the recent trend towards linguistic universal within the general community. In order to fill the gap, this paper investigates whether the established results in computational psycholinguistics can be generalized across languages. Specifically, we re-examine an established generalization -- the lower perplexity a language model has, the more human-like the language model is -- in Japanese with typologically different structures from English. Our experiments demonstrate that this established generalization exhibits a surprising lack of universality; namely, lower perplexity is not always human-like. Moreover, this discrepancy between English and Japanese is further explored from the perspective of (non-)uniform information density. Overall, our results suggest that a cross-lingual evaluation will be necessary to construct human-like computational models.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 12 of its 40 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

12 of 40 references.

References · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  9. 2020If beam search is the answer, what was the question?cited by 114
  10. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  11. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  12. 2021A Cognitive Regularizer for Language Modelingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 8; 2024: 12; 2025: 8; 2026: 12.

52 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  2. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  3. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  4. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  6. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  7. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  8. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  9. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  10. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  11. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
  12. 2025Large Language Models Are Human-Like Internallycited by 28
38 more
  1. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  2. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  3. 2022On the Effect of Anticipation on Reading Timescited by 18
  4. 2024On the Role of Context in Reading Time Predictioncited by 17
  5. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  6. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  7. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  8. 2022Probing BERT's priors with serial reproduction chainscited by 11
  9. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  10. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  11. 2024Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metricscited by 6
  12. 2021Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammarscited by 5
  13. 2022Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarizationcited by 5
  14. 2024Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysiscited by 5
  15. 2023Psychometric Predictive Power of Large Language Modelscited by 4
  16. 2025Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Modelcited by 4
  17. 2023Language Models are Drummers: Drum Composition with Natural Language Pre-Trainingcited by 3
  18. 2021Contextualized Semantic Distance between Highly Overlapped Textscited by 2
  19. 2022A Comparative Study on Textual Saliency of Styles from Eye Tracking, Annotations, and Language Modelscited by 2
  20. 2025A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behaviorcited by 2
  21. 2026Form and Meaning in Intrinsic Multilingual Evaluationscited by 2
  22. 2026Dual Alignment Between Language Model Layers and Human Sentence Processingcited by 2
  23. 2022Composition, Attention, or Both?cited by 1
  24. 2023Predicting Human Translation Difficulty with Neural Machine Translationcited by 1
  25. 2024Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differencescited by 1
  26. 2024Reverse-Engineering the Readercited by 1
  27. 2025Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptationcited by 1
  28. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  29. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 1
  30. 2023Style Locality for Controllable Generation with kNN Language Modelscited by 0
  31. 2025Detecting Spelling and Grammatical Anomalies in Russian Poetry Textscited by 0
  32. 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibilitycited by 0
  33. 2026Rethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMscited by 0
  34. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  35. 2026A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Modelscited by 0
  36. 2026K-Quantization and its Impact on Output Performancecited by 0
  37. 2026Timesteps of Mamba Align with Human Reading Timescited by 0
  38. 2026surprisal is Not a Theorycited by 0

50 most cited of 52 All 52 (JSON)