# Relating transformers to models and neural representations of the hippocampal formation arXiv: https://arxiv.org/abs/2112.04035 ยท HTML: https://mlsanity.com/paper/2112.04035/ ## Key facts - Cited by 62 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 38 references that are in the map. - First posted on arXiv 7 Dec 2021. ## Builds on (most cited first, 13 of 13) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020) - [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016) - [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016) - [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020) - [Learning to Encode Position for Transformer with Continuous Dynamical Model](https://mlsanity.com/paper/2003.09229/) (2020) - [Emergence of grid-like representations by training recurrent neural networks to perform spatial localization](https://mlsanity.com/paper/1803.07770/) (2018) - [R-Transformer: Recurrent Neural Network Enhanced Transformer](https://mlsanity.com/paper/1907.05572/) (2019) - [Hippocampal Spatial Mapping As Fast Graph Learning](https://mlsanity.com/paper/2107.00567/) (2021) ## Built on by (most cited first, 30 of 62) - [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022) - [Zamba: A Compact 7B SSM Hybrid Model](https://mlsanity.com/paper/2405.16712/) (2024) - [Evaluating Cognitive Maps and Planning in Large Language Models with CogEval](https://mlsanity.com/paper/2309.15129/) (2023) - [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022) - [Key-value memory in the brain](https://mlsanity.com/paper/2501.02950/) (2025) - [A Philosophical Introduction to Language Models - Part II: The Way Forward](https://mlsanity.com/paper/2405.03207/) (2024) - [Long Sequence Hopfield Memory](https://mlsanity.com/paper/2306.04532/) (2023) - [Metacognition is all you need? Using Introspection in Generative Agents to Improve Goal-directed Behavior](https://mlsanity.com/paper/2401.10910/) (2024) - [Linking In-context Learning in Transformers to Human Episodic Memory](https://mlsanity.com/paper/2405.14992/) (2024) - [Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences](https://mlsanity.com/paper/2509.16189/) (2025) - [Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models](https://mlsanity.com/paper/2309.16750/) (2023) - [Sparse and Structured Hopfield Networks](https://mlsanity.com/paper/2402.13725/) (2024) - [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025) - [Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data](https://mlsanity.com/paper/2311.00136/) (2023) - [Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks](https://mlsanity.com/paper/2410.08133/) (2024) - [System identification of neural systems: If we got it right, would we know?](https://mlsanity.com/paper/2302.06677/) (2023) - [MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models](https://mlsanity.com/paper/2507.20395/) (2025) - [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025) - [Graph schemas as abstractions for transfer learning, inference, and planning](https://mlsanity.com/paper/2302.07350/) (2023) - [Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective](https://mlsanity.com/paper/2509.09154/) (2025) - [V1T: large-scale mouse V1 response prediction using a Vision Transformer](https://mlsanity.com/paper/2302.03023/) (2023) - [The Multiscale Surface Vision Transformer](https://mlsanity.com/paper/2303.11909/) (2023) - [On Conformal Isometry of Grid Cells: Learning Distance-Preserving Position Embedding](https://mlsanity.com/paper/2405.16865/) (2024) - [Deep neural networks architectures from the perspective of manifold learning](https://mlsanity.com/paper/2306.03406/) (2023) - [Spatially-Aware Transformer for Embodied Agents](https://mlsanity.com/paper/2402.15160/) (2024) - [Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments](https://mlsanity.com/paper/2506.06981/) (2025) - [Deep Learning Models to Study Sentence Comprehension in the Human Brain](https://mlsanity.com/paper/2301.06340/) (2023) - [Probing neural representations of scene perception in a hippocampally dependent task using artificial neural networks](https://mlsanity.com/paper/2303.06367/) (2023) - [Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Tasks](https://mlsanity.com/paper/2402.08211/) (2024) - [A Scenario-Driven Cognitive Approach to Next-Generation AI Memory](https://mlsanity.com/paper/2509.13235/) (2025) ## Abstract (arXiv) Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.