# Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives

arXiv: https://arxiv.org/abs/2310.13676 · HTML: https://mlsanity.com/paper/2310.13676/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 100 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021)
- [What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability](https://mlsanity.com/paper/2305.11707/) (2023)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities](https://mlsanity.com/paper/2210.12187/) (2022)
- [A surprisal--duration trade-off across and within the world's languages](https://mlsanity.com/paper/2109.15000/) (2021)
- [Discourse structure interacts with reference but not syntax in neural language models](https://mlsanity.com/paper/2010.04887/) (2020)
- [A Cognitive Regularizer for Language Modeling](https://mlsanity.com/paper/2105.07144/) (2021)
- [Expectations over Unspoken Alternatives Predict Pragmatic Inferences](https://mlsanity.com/paper/2304.04758/) (2023)
- [Towards Pragmatic Production Strategies for Natural Language Generation Tasks](https://mlsanity.com/paper/2210.12828/) (2022)
- [Revisiting Entropy Rate Constancy in Text](https://mlsanity.com/paper/2305.12084/) (2023)

## Built on by (most cited first, 23 of 23)

- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2411.04075/) (2024)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse](https://mlsanity.com/paper/2410.16062/) (2024)
- [Towards a Similarity-adjusted Surprisal Theory](https://mlsanity.com/paper/2410.17676/) (2024)
- [The Harmonic Structure of Information Contours](https://mlsanity.com/paper/2506.03902/) (2025)
- [Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models](https://mlsanity.com/paper/2503.12072/) (2025)
- [Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?](https://mlsanity.com/paper/2602.14653/) (2026)
- [Decomposition of surprisal: Unified computational model of ERP components in language processing](https://mlsanity.com/paper/2409.06803/) (2024)
- [On the Proper Treatment of Units in Surprisal Theory](https://mlsanity.com/paper/2604.28147/) (2026)
- [Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs](https://mlsanity.com/paper/2505.02072/) (2025)
- [Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue](https://mlsanity.com/paper/2605.00506/) (2026)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models](https://mlsanity.com/paper/2602.00377/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Probing for Reading Times](https://mlsanity.com/paper/2604.18712/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models](https://mlsanity.com/paper/2610.13794/) (2026)

## Abstract (arXiv)

We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We introduce a method to obtain interpretable estimates of information value using neural text generators, and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. Information value is a stronger predictor of utterance acceptability in written and spoken dialogue than aggregates of token-level surprisal and it is complementary to surprisal for predicting eye-tracked reading times.