mlsanityData to 14 Oct 2026

Nonparametric Modern Hopfield Models

arXiv 2404.03900 ↗First posted 5 Apr 2024stat.ML cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We present a nonparametric interpretation for deep learning compatible modern Hopfield models and utilize this new perspective to debut efficient variants. Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs. Interestingly, our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient modern Hopfield models, by introducing textitsparse-structured modern Hopfield models with sub-quadratic complexity. We establish that this sparse model inherits the appealing theoretical properties of its dense analogue -- connection with transformer attention, fixed point convergence and exponential memory capacity. Additionally, we showcase the versatility of our framework by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-K and positive random feature modern Hopfield models. Empirically, we validate our framework in both synthetic and realistic settings for memory retrieval and learning tasks.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 36 of its 65 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

36 of 65 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  11. 2023BloombergGPT: A Large Language Model for Financecited by 939
  12. 2020Efficient Transformers: A Surveycited by 790
24 more
  1. 2020Hopfield Networks is All You Needcited by 475
  2. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  3. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  4. 2016Multiple Instance Learning: A Survey of Problem Characteristics and Applicationscited by 164
  5. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  6. 2019Adaptively Sparse Transformerscited by 155
  7. 2022On The Computational Complexity of Self-Attentioncited by 139
  8. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 112
  9. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  10. 2021Memory-efficient Transformers via Top-k Attentioncited by 60
  11. 2020Modern Hopfield Networks and Attention for Immune Repertoire Classificationcited by 53
  12. 2023Conformal Prediction for Time Series with Modern Hopfield Networkscited by 42
  13. 2023On Sparse Modern Hopfield Modelcited by 42
  14. 2024On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysiscited by 41
  15. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  16. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  17. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  18. 2023Context-enriched molecule representations improve few-shot drug discoverycited by 24
  19. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  20. 2023Why Softmax Attention Outperforms Linear Attentioncited by 15
  21. 2024Sparse and Structured Hopfield Networkscited by 13
  22. 2013Compact Random Feature Mapscited by 12
  23. 2022Kernel Memory Networks: A Unifying Framework for Memory Modelingcited by 12
  24. 2013An Equivalence between the Lasso and Support Vector Machinescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  2. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  3. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  4. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  5. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  6. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
  7. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  8. 2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
  9. 2024Modern Hopfield Networks Require Chain-of-Thought to Solve mathsfNC1-Hard Problemscited by 9
  10. 2024Dense Associative Memory Through the Lens of Random Featurescited by 8
  11. 2024Unlocking the Theory Behind Scaling 1-Bit Neural Networkscited by 4
  12. 2025Neural Algorithmic Reasoning for Hypergraphs with Looped Transformerscited by 4
4 more
  1. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  2. 2025Adaptive Hopfield Network: Rethinking Similarities in Associative Memorycited by 1
  3. 2025Modern Hopfield Networks with Continuous-Time Memoriescited by 0
  4. 2026Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamicscited by 0