# The Natural Stories Corpus

arXiv: https://arxiv.org/abs/1708.05763 · HTML: https://mlsanity.com/paper/1708.05763/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2017.

## Built on by (most cited first, 30 of 40)

- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens](https://mlsanity.com/paper/2304.11389/) (2023)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times](https://mlsanity.com/paper/2402.02255/) (2024)
- [How to Compute the Probability of a Word](https://mlsanity.com/paper/2406.14561/) (2024)
- [Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities](https://mlsanity.com/paper/2406.10851/) (2024)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [On the Effect of Anticipation on Reading Times](https://mlsanity.com/paper/2211.14301/) (2022)
- [Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal](https://mlsanity.com/paper/2212.11185/) (2022)
- [Linear Recency Bias During Training Improves Transformers' Fit to Reading Times](https://mlsanity.com/paper/2409.11250/) (2024)
- [Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?](https://mlsanity.com/paper/2310.17774/) (2023)
- [Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures](https://mlsanity.com/paper/2311.04547/) (2023)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [Analyzing Wrap-Up Effects through an Information-Theoretic Lens](https://mlsanity.com/paper/2203.17213/) (2022)
- [What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores](https://mlsanity.com/paper/2406.01538/) (2024)
- [Language Models Grow Less Humanlike beyond Phase Transition](https://mlsanity.com/paper/2502.18802/) (2025)
- [Towards a Similarity-adjusted Surprisal Theory](https://mlsanity.com/paper/2410.17676/) (2024)
- [Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?](https://mlsanity.com/paper/2311.09325/) (2023)
- [Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics](https://mlsanity.com/paper/2404.19178/) (2024)
- [The Impact of Token Granularity on the Predictive Power of Language Model Surprisal](https://mlsanity.com/paper/2412.11940/) (2024)
- [The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage](https://mlsanity.com/paper/2506.01172/) (2025)
- [Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research](https://mlsanity.com/paper/2509.22831/) (2025)
- [Psychometric Predictive Power of Large Language Models](https://mlsanity.com/paper/2311.07484/) (2023)
- [How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?](https://mlsanity.com/paper/2507.22209/) (2025)
- [Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models](https://mlsanity.com/paper/2508.05803/) (2025)
- [Strategic resource allocation in memory encoding: An efficiency principle shaping language processing](https://mlsanity.com/paper/2503.14728/) (2025)
- [Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Models](https://mlsanity.com/paper/2208.02402/) (2022)

## Abstract (arXiv)

It is now a common practice to compare models of human language processing by predicting participant reactions (such as reading times) to corpora consisting of rich naturalistic linguistic materials. However, many of the corpora used in these studies are based on naturalistic text and thus do not contain many of the low-frequency syntactic constructions that are often required to distinguish processing theories. Here we describe a new corpus consisting of English texts edited to contain many low-frequency syntactic constructions while still sounding fluent to native speakers. The corpus is annotated with hand-corrected parse trees and includes self-paced reading time data. Here we give an overview of the content of the corpus and release the data.