# Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings

arXiv: https://arxiv.org/abs/2507.05147 · HTML: https://mlsanity.com/paper/2507.05147/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 80 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Generalisation error in learning with random features and the hidden manifold model](https://mlsanity.com/paper/2002.09339/) (2020)
- [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://mlsanity.com/paper/2505.17638/) (2025)
- [Energy Transformer](https://mlsanity.com/paper/2302.07253/) (2023)
- [Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization](https://mlsanity.com/paper/2006.06560/) (2020)
- [In search of dispersed memories: Generative diffusion models are associative memory networks](https://mlsanity.com/paper/2309.17290/) (2023)
- [Inverse statistical problems: from the inverse Ising problem to data science](https://mlsanity.com/paper/1702.01522/) (2017)
- [Restricted Boltzmann Machine, recent advances and mean-field theory](https://mlsanity.com/paper/2011.11307/) (2020)
- [Learning through atypical "phase transitions" in overparameterized neural networks](https://mlsanity.com/paper/2110.00683/) (2021)
- [Mapping of attention mechanisms to a generalized Potts model](https://mlsanity.com/paper/2304.07235/) (2023)
- [Supervised Hebbian Learning](https://mlsanity.com/paper/2203.01304/) (2022)
- [Dreaming neural networks: forgetting spurious memories and reinforcing pure ones](https://mlsanity.com/paper/1810.12217/) (2018)
- [Tractability from overparametrization: The example of the negative perceptron](https://mlsanity.com/paper/2110.15824/) (2021)
- [Explaining the effects of non-convergent sampling in the training of Energy-Based Models](https://mlsanity.com/paper/2301.09428/) (2023)
- [Storage and Learning phase transitions in the Random-Features Hopfield Model](https://mlsanity.com/paper/2303.16880/) (2023)
- [Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks](https://mlsanity.com/paper/2201.00032/) (2021)
- [Fast training and sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2405.15376/) (2024)
- [Random Features Hopfield Networks generalize retrieval to previously unseen examples](https://mlsanity.com/paper/2407.05658/) (2024)
- [Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics](https://mlsanity.com/paper/2307.06797/) (2023)
- [Inferring Higher-Order Couplings with Neural Networks](https://mlsanity.com/paper/2501.06108/) (2025)
- [Bridging Associative Memory and Probabilistic Modeling](https://mlsanity.com/paper/2402.10202/) (2024)
- [A theoretical framework for overfitting in energy-based modeling](https://mlsanity.com/paper/2501.19158/) (2025)
- [Unsupervised hierarchical clustering using the learning dynamics of RBMs](https://mlsanity.com/paper/2302.01851/) (2023)
- [Self-attention as an attractor network: transient memories without backpropagation](https://mlsanity.com/paper/2409.16112/) (2024)
- [Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks](https://mlsanity.com/paper/2505.13230/) (2025)

## Built on by (most cited first, 2 of 2)

- [Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks](https://mlsanity.com/paper/2509.01349/) (2025)
- [Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data](https://mlsanity.com/paper/2604.26841/) (2026)

## Abstract (arXiv)

Energy-based probabilistic models learned by maximizing the likelihood of the data are limited by the intractability of the partition function. A widely used workaround is to maximize the pseudo-likelihood, which replaces the global normalization with tractable local normalizations. Here we show that, in the zero-temperature limit, a network trained to maximize pseudo-likelihood naturally implements an associative memory: if the training set is small, patterns become fixed-point attractors whose basins of attraction exceed those of any classical Hopfield rule. We explain quantitatively this effect on uncorrelated random patterns. Moreover, we show that, for different structured datasets coming from computer science (random feature model, MNIST), physics (spin glasses) and biology (proteins), as the number of training examples increases the learned network goes beyond memorization, developing meaningful attractors with non-trivial correlations with test examples, thus showing the ability to generalize. Our results therefore reveal pseudo-likelihood works both as an efficient inference tool and as a principled mechanism for memory and generalization.