# Information-Theoretic Storage Cost in Sentence Comprehension

arXiv: https://arxiv.org/abs/2602.18217 · HTML: https://mlsanity.com/paper/2602.18217/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 76 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [The Natural Stories Corpus](https://mlsanity.com/paper/1708.05763/) (2017)
- [Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times](https://mlsanity.com/paper/2402.02255/) (2024)
- [Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities](https://mlsanity.com/paper/2406.10851/) (2024)
- [What Can String Probability Tell Us About Grammaticality?](https://mlsanity.com/paper/2510.16227/) (2025)
- [Linguistic Dependencies and Statistical Dependence](https://mlsanity.com/paper/2104.08685/) (2021)
- [Strategic resource allocation in memory encoding: An efficiency principle shaping language processing](https://mlsanity.com/paper/2503.14728/) (2025)

## Built on by (most cited first, 1 of 1)

- [Syntactically-guided Information Maintenance in Sentence Comprehension](https://mlsanity.com/paper/2604.27468/) (2026)

## Abstract (arXiv)

Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have largely been formalized using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, probabilistic, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings and relative clauses, (ii) correlates with a grammar-based storage cost in a syntactically-annotated corpus, and (iii) predicts reading-time variance in two large-scale naturalistic datasets over and above baseline models with traditional information-based predictors. Our code is available at https://github.com/kohei-kaji/info-storage.