# Dreaming neural networks: forgetting spurious memories and reinforcing pure ones

arXiv: https://arxiv.org/abs/1810.12217 · HTML: https://mlsanity.com/paper/1810.12217/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 15 of 15)

- [Differentiable Physics: A Position Piece](https://mlsanity.com/paper/2109.07573/) (2021)
- [Random Features Hopfield Networks generalize retrieval to previously unseen examples](https://mlsanity.com/paper/2407.05658/) (2024)
- [Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting](https://mlsanity.com/paper/2308.01421/) (2023)
- [Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines](https://mlsanity.com/paper/2406.09924/) (2024)
- [Recurrent neural networks that generalize from examples and optimize by dreaming](https://mlsanity.com/paper/2204.07954/) (2022)
- [The Decimation Scheme for Symmetric Matrix Factorization](https://mlsanity.com/paper/2307.16564/) (2023)
- [Self-orthogonalizing attractor neural networks emerging from the free energy principle](https://mlsanity.com/paper/2505.22749/) (2025)
- [Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings](https://mlsanity.com/paper/2507.05147/) (2025)
- [Statistical Mechanics of Learning via Reverberation in Bidirectional Associative Memories](https://mlsanity.com/paper/2307.08365/) (2023)
- [On the role of non-linear latent features in bipartite generative neural networks](https://mlsanity.com/paper/2506.10552/) (2025)
- [Finite-size scaling of hetero-associative retrieval in continuous-signal-driven Ising spin systems](https://mlsanity.com/paper/2605.14059/) (2026)
- [Pavlov Learning Machines](https://mlsanity.com/paper/2207.00790/) (2022)
- [Ken Utilization Layer: Hebbian Replay Within a Student's Ken for Adaptive Exercise Recommendation](https://mlsanity.com/paper/2507.00032/) (2025)
- [Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamics](https://mlsanity.com/paper/2603.20656/) (2026)
- [Exponential Capacity in Multilayer Hetero-Associative Neural Networks](https://mlsanity.com/paper/2607.29554/) (2026)

## Abstract (arXiv)

The standard Hopfield model for associative neural networks accounts for biological Hebbian learning and acts as the harmonic oscillator for pattern recognition, however its maximal storage capacity is α ~ 0.14, far from the theoretical bound for symmetric networks, i.e. α =1. Inspired by sleeping and dreaming mechanisms in mammal brains, we propose an extension of this model displaying the standard on-line (awake) learning mechanism (that allows the storage of external information in terms of patterns) and an off-line (sleep) unlearning&consolidating mechanism (that allows spurious-pattern removal and pure-pattern reinforcement): this obtained daily prescription is able to saturate the theoretical bound α=1, remaining also extremely robust against thermal noise. Both neural and synaptic features are analyzed both analytically and numerically. In particular, beyond obtaining a phase diagram for neural dynamics, we focus on synaptic plasticity and we give explicit prescriptions on the temporal evolution of the synaptic matrix. We analytically prove that our algorithm makes the Hebbian kernel converge with high probability to the projection matrix built over the pure stored patterns. Furthermore, we obtain a sharp and explicit estimate for the "sleep rate" in order to ensure such a convergence. Finally, we run extensive numerical simulations (mainly Monte Carlo sampling) to check the approximations underlying the analytical investigations (e.g., we developed the whole theory at the so called replica-symmetric level, as standard in the Amit-Gutfreund-Sompolinsky reference framework) and possible finite-size effects, finding overall full agreement with the theory.