# Word class representations spontaneously emerge in a deep neural network trained on next word prediction

arXiv: https://arxiv.org/abs/2302.07588 · HTML: https://mlsanity.com/paper/2302.07588/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 83 references that are in the map.
- First posted on arXiv 15 Feb 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Cognitive computational neuroscience](https://mlsanity.com/paper/1807.11819/) (2018)
- [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018)
- [Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future](https://mlsanity.com/paper/2108.04543/) (2021)
- [A statistical method for analyzing and comparing spatiotemporal cortical activation patterns](https://mlsanity.com/paper/1611.07677/) (2016)
- [Analysis of structure and dynamics in three-neuron motifs](https://mlsanity.com/paper/1811.05225/) (2018)
- [Sparsity through evolutionary pruning prevents neuronal networks from overfitting](https://mlsanity.com/paper/1911.10988/) (2019)
- [Neural Network based Successor Representations of Space and Language](https://mlsanity.com/paper/2202.11190/) (2022)
- [Neural Network based Formation of Cognitive Maps of Semantic Spaces and the Emergence of Abstract Concepts](https://mlsanity.com/paper/2210.16062/) (2022)
- [Integration of Leaky-Integrate-and-Fire-Neurons in Deep Learning Architectures](https://mlsanity.com/paper/2004.13532/) (2020)
- [Classification at the Accuracy Limit -- Facing the Problem of Data Ambiguity](https://mlsanity.com/paper/2206.01922/) (2022)
- [Quantifying and maximizing the information flux in recurrent neural networks](https://mlsanity.com/paper/2301.12892/) (2023)

## Built on by (most cited first, 9 of 9)

- [Conceptual Cognitive Maps Formation with Neural Successor Networks and Word Embeddings](https://mlsanity.com/paper/2307.01577/) (2023)
- [Multi-Modal Cognitive Maps based on Neural Networks trained on Successor Representations](https://mlsanity.com/paper/2401.01364/) (2023)
- [Analysis of Argument Structure Constructions in a Deep Recurrent Language Model](https://mlsanity.com/paper/2408.03062/) (2024)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [Beyond Labels: Advancing Cluster Analysis with the Entropy of Distance Distribution (EDD)](https://mlsanity.com/paper/2311.16621/) (2023)
- [Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERT](https://mlsanity.com/paper/2405.02024/) (2024)
- [Probing for Consciousness in Machines](https://mlsanity.com/paper/2411.16262/) (2024)
- [Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems](https://mlsanity.com/paper/2603.29617/) (2026)
- [Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language](https://mlsanity.com/paper/2605.24585/) (2026)

## Abstract (arXiv)

How do humans learn language, and can the first language be learned at all? These fundamental questions are still hotly debated. In contemporary linguistics, there are two major schools of thought that give completely opposite answers. According to Chomsky's theory of universal grammar, language cannot be learned because children are not exposed to sufficient data in their linguistic environment. In contrast, usage-based models of language assume a profound relationship between language structure and language use. In particular, contextual mental processing and mental representations are assumed to have the cognitive capacity to capture the complexity of actual language use at all levels. The prime example is syntax, i.e., the rules by which words are assembled into larger units such as sentences. Typically, syntactic rules are expressed as sequences of word classes. However, it remains unclear whether word classes are innate, as implied by universal grammar, or whether they emerge during language acquisition, as suggested by usage-based approaches. Here, we address this issue from a machine learning and natural language processing perspective. In particular, we trained an artificial deep neural network on predicting the next word, provided sequences of consecutive words as input. Subsequently, we analyzed the emerging activation patterns in the hidden layers of the neural network. Strikingly, we find that the internal representations of nine-word input sequences cluster according to the word class of the tenth word to be predicted as output, even though the neural network did not receive any explicit information about syntactic rules or word classes during training. This surprising result suggests, that also in the human brain, abstract representational categories such as word classes may naturally emerge as a consequence of predictive coding and processing during language acquisition.