# Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERT arXiv: https://arxiv.org/abs/2405.02024 ยท HTML: https://mlsanity.com/paper/2405.02024/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 69 references that are in the map. - First posted on arXiv 3 May 2024. ## Builds on (most cited first, 26 of 26) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019) - [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020) - [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018) - [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019) - [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023) - [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023) - [Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks](https://mlsanity.com/paper/1801.07772/) (2018) - [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018) - [Universal Text Representation from BERT: An Empirical Study](https://mlsanity.com/paper/1910.07973/) (2019) - [On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets](https://mlsanity.com/paper/2109.03537/) (2021) - [A statistical method for analyzing and comparing spatiotemporal cortical activation patterns](https://mlsanity.com/paper/1611.07677/) (2016) - [Sparsity through evolutionary pruning prevents neuronal networks from overfitting](https://mlsanity.com/paper/1911.10988/) (2019) - [Neural Network based Successor Representations of Space and Language](https://mlsanity.com/paper/2202.11190/) (2022) - [Word class representations spontaneously emerge in a deep neural network trained on next word prediction](https://mlsanity.com/paper/2302.07588/) (2023) - [Neural Network based Formation of Cognitive Maps of Semantic Spaces and the Emergence of Abstract Concepts](https://mlsanity.com/paper/2210.16062/) (2022) - [Integration of Leaky-Integrate-and-Fire-Neurons in Deep Learning Architectures](https://mlsanity.com/paper/2004.13532/) (2020) - [Classification at the Accuracy Limit -- Facing the Problem of Data Ambiguity](https://mlsanity.com/paper/2206.01922/) (2022) - [pylustrator: Code generation for reproducible figures for publication](https://mlsanity.com/paper/1910.00279/) (2019) - [Multi-Modal Cognitive Maps based on Neural Networks trained on Successor Representations](https://mlsanity.com/paper/2401.01364/) (2023) - [Beyond Labels: Advancing Cluster Analysis with the Entropy of Distance Distribution (EDD)](https://mlsanity.com/paper/2311.16621/) (2023) ## Built on by (most cited first, 1 of 1) - [Analysis of Argument Structure Constructions in a Deep Recurrent Language Model](https://mlsanity.com/paper/2408.03062/) (2024) ## Abstract (arXiv) The ability to transmit and receive complex information via language is unique to humans and is the basis of traditions, culture and versatile social interactions. Through the disruptive introduction of transformer based large language models (LLMs) humans are not the only entity to "understand" and produce language any more. In the present study, we have performed the first steps to use LLMs as a model to understand fundamental mechanisms of language processing in neural networks, in order to make predictions and generate hypotheses on how the human brain does language processing. Thus, we have used ChatGPT to generate seven different stylistic variations of ten different narratives (Aesop's fables). We used these stories as input for the open source LLM BERT and have analyzed the activation patterns of the hidden units of BERT using multi-dimensional scaling and cluster analysis. We found that the activation vectors of the hidden units cluster according to stylistic variations in earlier layers of BERT (1) than narrative content (4-5). Despite the fact that BERT consists of 12 identical building blocks that are stacked and trained on large text corpora, the different layers perform different tasks. This is a very useful model of the human brain, where self-similar structures, i.e. different areas of the cerebral cortex, can have different functions and are therefore well suited to processing language in a very efficient way. The proposed approach has the potential to open the black box of LLMs on the one hand, and might be a further step to unravel the neural processes underlying human language processing and cognition in general.