# Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal

arXiv: https://arxiv.org/abs/2606.05346 · HTML: https://mlsanity.com/paper/2606.05346/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 25 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 3 of 3)

- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens](https://mlsanity.com/paper/2304.11389/) (2023)

## Built on by (most cited first, 3 of 3)

- [surprisal is Not a Theory](https://mlsanity.com/paper/2607.20208/) (2026)
- [A Hierarchical Energy-Based Model for Multimodal Cognition](https://mlsanity.com/paper/2608.12398/) (2026)
- [Evaluating Trajectory Features for Routing Final-Layer Attention](https://mlsanity.com/paper/2610.09272/) (2026)

## Abstract (arXiv)

Human language comprehension unfolds sequentially: each word is processed in the context of those that came before, and the interpretation builds incrementally over time. Surprisal, the negative log probability of a word given its context, has been the dominant predictor of incremental processing cost. But surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Dynamical-systems approaches suggest that the trajectory of the evolving interpretive state, not just its position at each moment,should shape processing, and language itself may have local momentum, since speakers plan utterances a few words at a time. We introduce trajectory extrapolation error: at each word, we fit a linear trajectory to the preceding hidden states of a transformer language model and measure deviation from the extrapolated path. On the Natural Stories corpus, this measure is nearly orthogonal to surprisal (r = .044) and independently predicts self-paced reading times. The effect is especially pronounced in garden-path sentences, strengthens with model scale (GPT-2 Small to Large), and replicates across architectures with different positional encoding schemes (GPT-2 vs. Pythia/RoPE). A displacement control shows the effect is not reducible to representational change magnitude: displacement and extrapolation error predict in opposite directions. These findings reveal two dissociable components of processing cost: word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error).