# Long Sequence Hopfield Memory arXiv: https://arxiv.org/abs/2306.04532 ยท HTML: https://mlsanity.com/paper/2306.04532/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 83 references that are in the map. - First posted on arXiv 7 Jun 2023. ## Builds on (most cited first, 23 of 23) - [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021) - [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020) - [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015) - [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023) - [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023) - [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016) - [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020) - [Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization](https://mlsanity.com/paper/1802.01569/) (2018) - [Relating transformers to models and neural representations of the hippocampal formation](https://mlsanity.com/paper/2112.04035/) (2021) - [Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortex](https://mlsanity.com/paper/1511.00083/) (2015) - [Dense Associative Memory is Robust to Adversarial Inputs](https://mlsanity.com/paper/1701.00939/) (2017) - [Hierarchical Associative Memory](https://mlsanity.com/paper/2107.06446/) (2021) - [Recurrent Network Models Of Sequence Generation And Memory](https://mlsanity.com/paper/1603.04687/) (2016) - [Simplicial Hopfield networks](https://mlsanity.com/paper/2305.05179/) (2023) - [Sequential Memory with Temporal Predictive Coding](https://mlsanity.com/paper/2305.11982/) (2023) - [Activation function dependence of the storage capacity of treelike neural networks](https://mlsanity.com/paper/2007.11136/) (2020) - [Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning](https://mlsanity.com/paper/2211.14067/) (2022) - [Metastable attractors explain the variable timing of stable behavioral action sequences](https://mlsanity.com/paper/2001.09600/) (2020) - [Neural mechanisms underlying the temporal organization of naturalistic animal behavior](https://mlsanity.com/paper/2203.02151/) (2022) - [Spectrum of non-Hermitian deep-Hebbian neural networks](https://mlsanity.com/paper/2208.11411/) (2022) - [Robust retrieval of dynamic sequences through interaction modulation](https://mlsanity.com/paper/2211.17152/) (2022) - [On neural network kernels and the storage capacity problem](https://mlsanity.com/paper/2201.04669/) (2022) - [Energy-based General Sequential Episodic Memory Networks at the Adiabatic Limit](https://mlsanity.com/paper/2212.05563/) (2022) ## Built on by (most cited first, 16 of 16) - [Outlier-Efficient Hopfield Layers for Large Transformer-Based Models](https://mlsanity.com/paper/2404.03828/) (2024) - [BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model](https://mlsanity.com/paper/2404.03830/) (2024) - [Dense Associative Memory Through the Lens of Random Features](https://mlsanity.com/paper/2410.24153/) (2024) - [Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks](https://mlsanity.com/paper/2411.05849/) (2024) - [Learning Sequence Attractors in Recurrent Networks with Hidden Neurons](https://mlsanity.com/paper/2404.02729/) (2024) - [Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimization](https://mlsanity.com/paper/2604.05042/) (2026) - [Episodic Memory Theory for the Mechanistic Interpretation of Recurrent Neural Networks](https://mlsanity.com/paper/2310.02430/) (2023) - [Modern Hopfield Networks meet Encoded Neural Representations -- Addressing Practical Considerations](https://mlsanity.com/paper/2409.16408/) (2024) - [Predictive Attractor Models](https://mlsanity.com/paper/2410.02430/) (2024) - [A Neural Model of Rule Discovery with Relatively Short-Term Sequence Memory](https://mlsanity.com/paper/2412.06839/) (2024) - [A General Close-loop Predictive Coding Framework for Auditory Working Memory](https://mlsanity.com/paper/2503.12506/) (2025) - [Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionals](https://mlsanity.com/paper/2507.01052/) (2025) - [Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamics](https://mlsanity.com/paper/2603.20656/) (2026) - [Beyond Fixed Points: Superpolynomial Capacity of Asymmetric Hopfield Networks](https://mlsanity.com/paper/2605.24611/) (2026) - [Neural Structural Reasoner: A Brain-inspired Architecture for Reasoning over Structured Knowledge](https://mlsanity.com/paper/2609.36620/) (2026) - [DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time](https://mlsanity.com/paper/2610.15185/) (2026) ## Abstract (arXiv) Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, and verify these theoretical results with numerical simulation. Moreover, we introduce a generalized pseudoinverse rule to recall sequences of highly correlated patterns. Finally, we extend this model to store sequences with variable timing between states' transitions and describe a biologically-plausible implementation, with connections to motor neuroscience.