mlsanityData to 7 Oct 2026

Relating transformers to models and neural representations of the hippocampal formation

arXiv 2112.04035 ↗First posted 7 Dec 2021cs.NE cs.LG q-bio.NCcited by 62

Not on the map. Why?

Abstract (arXiv)

Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 13 of its 38 references that are in the map. First posted on arXiv 7 Dec 2021.

Builds on

13 of 38 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Hopfield Networks is All You Needcited by 461
  7. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  8. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  9. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  10. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  11. 2018Emergence of grid-like representations by training recurrent neural networks to perform spatial localizationcited by 52
  12. 2019R-Transformer: Recurrent Neural Network Enhanced Transformercited by 39
1 more
  1. 2021Hippocampal Spatial Mapping As Fast Graph Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 19; 2024: 16; 2025: 16; 2026: 7.

62 papers.

Sort:

By area:

  1. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  2. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  3. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  4. 2022Disentanglement with Biological Constraints: A Theory of Functional Cell Typescited by 30
  5. 2025Key-value memory in the braincited by 22
  6. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
  7. 2023Long Sequence Hopfield Memorycited by 15
  8. 2024Metacognition is all you need? Using Introspection in Generative Agents to Improve Goal-directed Behaviorcited by 15
  9. 2024Linking In-context Learning in Transformers to Human Episodic Memorycited by 15
  10. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 14
  11. 2024Sparse and Structured Hopfield Networkscited by 13
  12. 2023Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Modelscited by 12
38 more
  1. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  2. 2023Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Datacited by 10
  3. 2023System identification of neural systems: If we got it right, would we know?cited by 9
  4. 2024Assessing Episodic Memory in LLMs with Sequence Order Recall Taskscited by 9
  5. 2025MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Modelscited by 8
  6. 2023Graph schemas as abstractions for transfer learning, inference, and planningcited by 7
  7. 2025Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspectivecited by 7
  8. 2023The Multiscale Surface Vision Transformercited by 6
  9. 2024On Conformal Isometry of Grid Cells: Learning Distance-Preserving Position Embeddingcited by 6
  10. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  11. 2023V1T: large-scale mouse V1 response prediction using a Vision Transformercited by 5
  12. 2023Deep neural networks architectures from the perspective of manifold learningcited by 5
  13. 2024Spatially-Aware Transformer for Embodied Agentscited by 5
  14. 2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
  15. 2023Deep Learning Models to Study Sentence Comprehension in the Human Braincited by 3
  16. 2023Probing neural representations of scene perception in a hippocampally dependent task using artificial neural networkscited by 3
  17. 2024Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Taskscited by 3
  18. 2025A Scenario-Driven Cognitive Approach to Next-Generation AI Memorycited by 3
  19. 2023Emergence of Grid-like Representations by Training Recurrent Networks with Conformal Normalizationcited by 2
  20. 2024Semantically-correlated memories in a dense associative modelcited by 2
  21. 2025Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamicscited by 2
  22. 2025Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Taskscited by 2
  23. 2022Grid cells and their potential application in AIcited by 1
  24. 2022Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networkscited by 1
  25. 2023Building artificial neural circuits for domain-general cognition: a primer on brain-inspired systems-level architecturecited by 1
  26. 2023Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticitycited by 1
  27. 2023Hopfield-Enhanced Deep Neural Networks for Artifact-Resilient Brain State Decodingcited by 1
  28. 2024Quality Diversity for Robot Learning: Limitations and Future Directionscited by 1
  29. 2024Decoding Linguistic Representations of Human Braincited by 1
  30. 2024The Phenomenology of Machine: A Comprehensive Analysis of the Sentience of the OpenAI-o1 Model Integrating Functionalism, Consciousness Theories, Active Inference, and AI Architecturescited by 1
  31. 2024Deep reinforcement learning with time-scale invariant memorycited by 1
  32. 2025Building spatial world models from sparse transitional episodic memoriescited by 1
  33. 2025From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformerscited by 1
  34. 2025On the Role of Hidden States of Modern Hopfield Network in Transformercited by 1
  35. 2026ReCoN-Ipsundrum: An Inspectable Recurrent Persistence Loop Agent with Affect-Coupled Control and Mechanism-Linked Consciousness Indicator Assayscited by 1
  36. 2026SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMscited by 1
  37. 2023Implementing engrams from a machine learning perspective: matching for predictioncited by 0
  38. 2023Beyond Transformers for Function Learningcited by 0

50 most cited of 62 All 62 (JSON)