mlsanityData to 14 Oct 2026

Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned

arXiv 2410.12057 ↗First posted 15 Oct 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer are typically better estimators of human productions, but they reliably under-estimate the probabilities of human responses, over-rank rare responses, under-rank top responses, and produce highly distinct semantic spaces. Altogether, this work demonstrates in a tractable, interpretable domain that LM generations can not be used as replacements of or models of the cloze task.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 20 of its 49 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

20 of 49 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  7. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  9. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  10. 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
  11. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  12. 2020Topic Modeling with Contextualized Word Representation Clusterscited by 41
8 more
  1. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  2. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  3. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  4. 2024Generalized Measures of Anticipation and Responsivity in Online Language Processingcited by 24
  5. 2023Humans and language models diverge when predicting repeating textcited by 13
  6. 2023Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?cited by 13
  7. 2024The Effect of Surprisal on Reading Times in Information Seeking and Repeated Readingcited by 8
  8. 2024Bayesian Statistical Modeling with Predictors from LLMscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Setscited by 2
  2. 2026Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisalcited by 1
  3. 2026Can LLMs capture stable human-generated sentence entropy measures?cited by 1
  4. 2026Measuring Form and Function in Language Modelscited by 1
  5. 2026The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scalecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
  2. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?
  3. 2024Evil twins are not that evil: Qualitative insights into machine-generated prompts
  4. 2024Better & Faster Large Language Models via Multi-token Prediction
  5. 2026Beyond Confidence: The Rhythms of Reasoning in Generative Models
  6. 2025Alternatives To Next Token Prediction In Text Generation -- A Survey
  7. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
  8. 2025Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Models