mlsanityData to 7 Oct 2026

Long Sequence Hopfield Memory

arXiv 2306.04532 ↗First posted 7 Jun 2023cs.NE cond-mat.dis-nn cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, and verify these theoretical results with numerical simulation. Moreover, we introduce a generalized pseudoinverse rule to recall sequences of highly correlated patterns. Finally, we extend this model to store sequences with variable timing between states' transitions and describe a biologically-plausible implementation, with connections to motor neuroscience.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 23 of its 83 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

23 of 83 references.

References · 23

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2020Hopfield Networks is All You Needcited by 461
  3. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  6. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  7. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  8. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  9. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  10. 2017Dense Associative Memory is Robust to Adversarial Inputscited by 45
  11. 2015Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortexcited by 44
  12. 2021Hierarchical Associative Memorycited by 34
11 more
  1. 2016Recurrent Network Models Of Sequence Generation And Memorycited by 21
  2. 2023Simplicial Hopfield networkscited by 16
  3. 2023Sequential Memory with Temporal Predictive Codingcited by 14
  4. 2020Activation function dependence of the storage capacity of treelike neural networkscited by 10
  5. 2022Dense Hebbian neural networks: a replica symmetric picture of unsupervised learningcited by 10
  6. 2020Metastable attractors explain the variable timing of stable behavioral action sequencescited by 4
  7. 2022Neural mechanisms underlying the temporal organization of naturalistic animal behaviorcited by 4
  8. 2022Spectrum of non-Hermitian deep-Hebbian neural networkscited by 4
  9. 2022Robust retrieval of dynamic sequences through interaction modulationcited by 3
  10. 2022On neural network kernels and the storage capacity problemcited by 2
  11. 2022Energy-based General Sequential Episodic Memory Networks at the Adiabatic Limitcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 2; 2026: 4.

15 papers.

Sort:

  1. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  2. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  3. 2024Dense Associative Memory Through the Lens of Random Featurescited by 8
  4. 2024Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networkscited by 5
  5. 2024Learning Sequence Attractors in Recurrent Networks with Hidden Neuronscited by 1
  6. 2026Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimizationcited by 1
  7. 2023Episodic Memory Theory for the Mechanistic Interpretation of Recurrent Neural Networkscited by 0
  8. 2024Modern Hopfield Networks meet Encoded Neural Representations -- Addressing Practical Considerationscited by 0
  9. 2024Predictive Attractor Modelscited by 0
  10. 2024A Neural Model of Rule Discovery with Relatively Short-Term Sequence Memorycited by 0
  11. 2025A General Close-loop Predictive Coding Framework for Auditory Working Memorycited by 0
  12. 2025Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionalscited by 0
3 more
  1. 2026Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamicscited by 0
  2. 2026Beyond Fixed Points: Superpolynomial Capacity of Asymmetric Hopfield Networkscited by 0
  3. 2026Neural Structural Reasoner: A Brain-inspired Architecture for Reasoning over Structured Knowledgecited by 0