# Analysis of Argument Structure Constructions in a Deep Recurrent Language Model

arXiv: https://arxiv.org/abs/2408.03062 · HTML: https://mlsanity.com/paper/2408.03062/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 68 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Learning Natural Language Inference with LSTM](https://mlsanity.com/paper/1512.08849/) (2015)
- [Word Embeddings: A Survey](https://mlsanity.com/paper/1901.09069/) (2019)
- [Cognitive computational neuroscience](https://mlsanity.com/paper/1807.11819/) (2018)
- [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018)
- [A statistical method for analyzing and comparing spatiotemporal cortical activation patterns](https://mlsanity.com/paper/1611.07677/) (2016)
- [Analysis of structure and dynamics in three-neuron motifs](https://mlsanity.com/paper/1811.05225/) (2018)
- [Word class representations spontaneously emerge in a deep neural network trained on next word prediction](https://mlsanity.com/paper/2302.07588/) (2023)
- [Classification at the Accuracy Limit -- Facing the Problem of Data Ambiguity](https://mlsanity.com/paper/2206.01922/) (2022)
- [Quantifying and maximizing the information flux in recurrent neural networks](https://mlsanity.com/paper/2301.12892/) (2023)
- [Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERT](https://mlsanity.com/paper/2405.02024/) (2024)

## Built on by (most cited first, 3 of 3)

- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems](https://mlsanity.com/paper/2603.29617/) (2026)
- [A Hierarchical and Attentional Analysis of Argument Structure Constructions in BERT Using Naturalistic Corpora](https://mlsanity.com/paper/2602.00554/) (2026)

## Abstract (arXiv)

Understanding how language and linguistic constructions are processed in the brain is a fundamental question in cognitive computational neuroscience. In this study, we explore the representation and processing of Argument Structure Constructions (ASCs) in a recurrent neural language model. We trained a Long Short-Term Memory (LSTM) network on a custom-made dataset consisting of 2000 sentences, generated using GPT-4, representing four distinct ASCs: transitive, ditransitive, caused-motion, and resultative constructions. We analyzed the internal activations of the LSTM model's hidden layers using Multidimensional Scaling (MDS) and t-Distributed Stochastic Neighbor Embedding (t-SNE) to visualize the sentence representations. The Generalized Discrimination Value (GDV) was calculated to quantify the degree of clustering within these representations. Our results show that sentence representations form distinct clusters corresponding to the four ASCs across all hidden layers, with the most pronounced clustering observed in the last hidden layer before the output layer. This indicates that even a relatively simple, brain-constrained recurrent neural network can effectively differentiate between various construction types. These findings are consistent with previous studies demonstrating the emergence of word class and syntax rule representations in recurrent language models trained on next word prediction tasks. In future work, we aim to validate these results using larger language models and compare them with neuroimaging data obtained during continuous speech perception. This study highlights the potential of recurrent neural language models to mirror linguistic processing in the human brain, providing valuable insights into the computational and neural mechanisms underlying language understanding.