# Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language

arXiv: https://arxiv.org/abs/2605.24585 · HTML: https://mlsanity.com/paper/2605.24585/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 63 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [Neural Network based Successor Representations of Space and Language](https://mlsanity.com/paper/2202.11190/) (2022)
- [Word class representations spontaneously emerge in a deep neural network trained on next word prediction](https://mlsanity.com/paper/2302.07588/) (2023)
- [Neural Network based Formation of Cognitive Maps of Semantic Spaces and the Emergence of Abstract Concepts](https://mlsanity.com/paper/2210.16062/) (2022)
- [Conceptual Cognitive Maps Formation with Neural Successor Networks and Word Embeddings](https://mlsanity.com/paper/2307.01577/) (2023)
- [Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems](https://mlsanity.com/paper/2603.29617/) (2026)

## Abstract (arXiv)

Neural language models are typically trained on next-token prediction, although linguistic structure spans multiple temporal scales. Successor representations (SRs) make this horizon explicit by encoding discounted distributions over future states. Here, we ask whether such predictive representations can recover not only word classes, but also finer functional and construction-like structure from natural language. A residual network trained on WikiText-103 predicts SR distributions at three horizons without part-of-speech supervision. At the shortest horizon, unsupervised clustering robustly recovers nouns, verbs, and adjectives, while directed inter-cluster transitions reproduce familiar syntactic asymmetries. At finer resolutions and across 13 part-of-speech categories, the same geometry reveals semantic-functional groupings that cross category boundaries and directed relations tracing candidate date, measurement, and title-name constructions. Part-of-speech agreement declines as the predictive horizon lengthens. These results suggest that word classes are coarse regions within a richer predictive geometry in which categorical and construction-like linguistic structure emerge from future-word distributions.