# Energy-Based Transformers as Predictors of Reading Difficulty

arXiv: https://arxiv.org/abs/2606.23382 · HTML: https://mlsanity.com/paper/2606.23382/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 34 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 4 of 4)

- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Modern Methods in Associative Memory](https://mlsanity.com/paper/2507.06211/) (2025)
- [NRGPT: An Energy-based Alternative for GPT](https://mlsanity.com/paper/2512.16762/) (2025)
- [N-gram-like Language Models Predict Naturalistic Reading Time Best](https://mlsanity.com/paper/2603.09872/) (2026)

## Abstract (arXiv)

Transformer language models have become established tools for modeling human sentence processing, with measures such as surprisal and attention entropy serving as effective predictors of reading difficulty that together capture complementary aspects of processing load. Here, we explore a related class of transformer models: energy-based transformers, which provide a principled formal link to associative memory models, bringing processing research into direct contact with the broader literature on Hopfield networks and dense associative memory. To our knowledge, this is the first exploration of an energy-based transformer measure in computational psycholinguistics. Across reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading), the energy measure is a robust predictor of reading times, providing significant fit beyond surprisal and entropy in all three. In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry. We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.