mlsanityData to 7 Oct 2026

Hopfield Networks is All You Need

arXiv 2008.02217 ↗First posted 16 Jul 2020cs.NE cs.CL cs.LGcited by 461

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors. It has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. The new update rule is equivalent to the attention mechanism used in transformers. This equivalence enables a characterization of the heads of transformer models. These heads perform in the first layers preferably global averaging and in higher layers partial averaging via metastable states. The new modern Hopfield network can be integrated into deep learning architectures as layers to allow the storage of and access to raw input data, intermediate results, or learned prototypes. These Hopfield layers enable new ways of deep learning, beyond fully-connected, convolutional, or recurrent networks, and provide pooling, memory, association, and attention mechanisms. We demonstrate the broad applicability of the Hopfield layers across various domains. Hopfield layers improved state-of-the-art on three out of four considered multiple instance learning problems as well as on immune repertoire classification with several hundreds of thousands of instances. On the UCI benchmark collections of small classification tasks, where deep learning methods typically struggle, Hopfield layers yielded a new state-of-the-art when compared to different machine learning methods. Finally, Hopfield layers achieved state-of-the-art on two drug design datasets. The implementation is available at: https://github.com/ml-jku/hopfield-layers

Key facts. Cited by 461 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 45; 2023: 67; 2024: 82; 2025: 96; 2026: 140.

461 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2022Transformers learn in-context by gradient descentcited by 714
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  5. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  6. 2022Long Range Language Modeling via Gated State Spacescited by 265
  7. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  8. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  9. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  10. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  11. 2022Liquid Structural State-Space Modelscited by 134
  12. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
38 more
  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  3. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidancecited by 108
  4. 2021On the Adversarial Robustness of Vision Transformerscited by 104
  5. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  6. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  7. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  8. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  9. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
  10. 2023Energy Transformercited by 69
  11. 2021A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challengescited by 68
  12. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  13. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  14. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  15. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  16. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 61
  17. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  18. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  19. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 55
  20. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 52
  21. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  22. 2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
  23. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 45
  24. 2023In search of dispersed memories: Generative diffusion models are associative memory networkscited by 44
  25. 2021Associative Memories via Predictive Codingcited by 42
  26. 2023On Sparse Modern Hopfield Modelcited by 41
  27. 2023Conformal Prediction for Time Series with Modern Hopfield Networkscited by 40
  28. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  29. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 39
  30. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)cited by 39
  31. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  32. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  33. 2023The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instabilitycited by 37
  34. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  35. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  36. 2021Hierarchical Associative Memorycited by 34
  37. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34
  38. 2024Spatiotemporal Skip Guidance for Enhanced Video Diffusion Samplingcited by 33

50 most cited of 461 All 461 (JSON)