# Analysis of Argument Structure Constructions in the Large Language Model BERT arXiv: https://arxiv.org/abs/2408.04270 ยท HTML: https://mlsanity.com/paper/2408.04270/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 84 references that are in the map. - First posted on arXiv 8 Aug 2024. ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016) - [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019) - [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019) - [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019) - [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019) - [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024) - [Learning Natural Language Inference with LSTM](https://mlsanity.com/paper/1512.08849/) (2015) - [The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative](https://mlsanity.com/paper/2210.13181/) (2022) - [CxGBERT: BERT meets Construction Grammar](https://mlsanity.com/paper/2011.04134/) (2020) - [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018) - [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022) - [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023) - [The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT](https://mlsanity.com/paper/2101.09115/) (2021) - [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023) - [A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces](https://mlsanity.com/paper/2305.18598/) (2023) - [How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure](https://mlsanity.com/paper/2311.04900/) (2023) - [Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons](https://mlsanity.com/paper/2403.17760/) (2024) - [A statistical method for analyzing and comparing spatiotemporal cortical activation patterns](https://mlsanity.com/paper/1611.07677/) (2016) - [Analysis of structure and dynamics in three-neuron motifs](https://mlsanity.com/paper/1811.05225/) (2018) - [How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT's Attention](https://mlsanity.com/paper/2011.00943/) (2020) - [Word class representations spontaneously emerge in a deep neural network trained on next word prediction](https://mlsanity.com/paper/2302.07588/) (2023) - [Classification at the Accuracy Limit -- Facing the Problem of Data Ambiguity](https://mlsanity.com/paper/2206.01922/) (2022) - [Quantifying and maximizing the information flux in recurrent neural networks](https://mlsanity.com/paper/2301.12892/) (2023) - [Analysis of Argument Structure Constructions in a Deep Recurrent Language Model](https://mlsanity.com/paper/2408.03062/) (2024) - [Enhancing Language Representation with Constructional Information for Natural Language Understanding](https://mlsanity.com/paper/2306.02819/) (2023) ## Built on by (most cited first, 3 of 3) - [Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment](https://mlsanity.com/paper/2510.13698/) (2025) - [Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems](https://mlsanity.com/paper/2603.29617/) (2026) - [Do Large Language Models know who did what to whom?](https://mlsanity.com/paper/2504.16884/) (2025) ## Abstract (arXiv) This study investigates how BERT processes and represents Argument Structure Constructions (ASCs), extending previous LSTM analyses. Using a dataset of 2000 sentences across four ASC types (transitive, ditransitive, caused-motion, resultative), we analyzed BERT's token embeddings across 12 layers. Visualizations with MDS and t-SNE and clustering quantified by Generalized Discrimination Value (GDV) were used. Feedforward classifiers (probes) predicted construction categories from embeddings. CLS token embeddings clustered best in layers 2-4, decreased in intermediate layers, and slightly increased in final layers. DET and SUBJ embeddings showed consistent clustering in intermediate layers, VERB embeddings increased in clustering from layer 1 to 12, and OBJ embeddings peaked in layer 10. Probe accuracies indicated low construction information in layer 1, with over 90 percent accuracy from layer 2 onward, revealing latent construction information beyond GDV clustering. Fisher Discriminant Ratio (FDR) analysis of attention weights showed OBJ tokens were crucial for differentiating ASCs, followed by VERB and DET tokens. SUBJ, CLS, and SEP tokens had insignificant FDR scores. This study highlights BERT's layered processing of linguistic constructions and its differences from LSTMs. Future research will compare these findings with neuroimaging data to understand the neural correlates of ASC processing. This research underscores neural language models' potential to mirror linguistic processing in the human brain, offering insights into the computational and neural mechanisms underlying language understanding.