mlsanityData to 14 Oct 2026

Sparse and Structured Hopfield Networks

arXiv 2402.13725 ↗First posted 21 Feb 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 20 of its 51 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

20 of 51 references.

References · 20

  1. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  2. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  3. 2020Hopfield Networks is All You Needcited by 475
  4. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  5. 2016Rationalizing Neural Predictionscited by 281
  6. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  7. 2019Sparse Sequence-to-Sequence Modelscited by 170
  8. 2019Adaptively Sparse Transformerscited by 155
  9. 2019Interpretable Neural Predictions with Differentiable Binary Variablescited by 126
  10. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  11. 2019Learning with Fenchel-Young Lossescited by 119
  12. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79
8 more
  1. 2018SparseMAP: Differentiable Sparse Structured Inferencecited by 62
  2. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  3. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 55
  4. 2023On Sparse Modern Hopfield Modelcited by 42
  5. 2017A Regularized Framework for Sparse and Structured Neural Attentioncited by 38
  6. 2021Explain and Predict, and then Predict Againcited by 29
  7. 2021SPECTRA: Sparse Structured Text Rationalizationcited by 19
  8. 2021Biological learning in key-value memory networkscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 2.

13 papers.

Sort:

  1. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  2. 2024Nonparametric Modern Hopfield Modelscited by 16
  3. 2025Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generationcited by 13
  4. 2025Dense Associative Memory with Epanechnikov Energycited by 11
  5. 2024Dense Associative Memory Through the Lens of Random Featurescited by 8
  6. 2025In-context denoising with one-layer transformers: connections between attention and associative memory retrievalcited by 7
  7. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  8. 2025Dense associative memory for Gaussian distributionscited by 2
  9. 2025Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Taskscited by 2
  10. 2026Sparse Attention as Compact Kernel Regressioncited by 2
  11. 2025Modern Hopfield Networks with Continuous-Time Memoriescited by 0
  12. 2025Transformer Circuits Can Realize Clustering Algorithmscited by 0
1 more
  1. 2026Hyperbolic Neural Population Geometry Benefits Computationcited by 0