# Recurrent neural networks that generalize from examples and optimize by dreaming

arXiv: https://arxiv.org/abs/2204.07954 · HTML: https://mlsanity.com/paper/2204.07954/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 51 references that are in the map.
- First posted on arXiv 17 Apr 2022.

## Builds on (most cited first, 10 of 10)

- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Phase transitions in Restricted Boltzmann Machines with generic priors](https://mlsanity.com/paper/1612.03132/) (2016)
- [The emergence of a concept in shallow neural networks](https://mlsanity.com/paper/2109.00454/) (2021)
- [Dreaming neural networks: forgetting spurious memories and reinforcing pure ones](https://mlsanity.com/paper/1810.12217/) (2018)
- [The backtracking survey propagation algorithm for solving random K-SAT problems](https://mlsanity.com/paper/1508.05117/) (2015)
- [Neural networks with redundant representation: detecting the undetectable](https://mlsanity.com/paper/1911.12689/) (2019)
- [Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines](https://mlsanity.com/paper/2107.06013/) (2021)
- [Dreaming neural networks: rigorous results](https://mlsanity.com/paper/1812.09077/) (2018)

## Built on by (most cited first, 2 of 2)

- [Phase transitions in the mini-batch size for sparse and dense two-layer neural networks](https://mlsanity.com/paper/2305.06435/) (2023)
- [Pavlov Learning Machines](https://mlsanity.com/paper/2207.00790/) (2022)

## Abstract (arXiv)

The gap between the huge volumes of data needed to train artificial neural networks and the relatively small amount of data needed by their biological counterparts is a central puzzle in machine learning. Here, inspired by biological information-processing, we introduce a generalized Hopfield network where pairwise couplings between neurons are built according to Hebb's prescription for on-line learning and allow also for (suitably stylized) off-line sleeping mechanisms. Moreover, in order to retain a learning framework, here the patterns are not assumed to be available, instead, we let the network experience solely a dataset made of a sample of noisy examples for each pattern. We analyze the model by statistical-mechanics tools and we obtain a quantitative picture of its capabilities as functions of its control parameters: the resulting network is an associative memory for pattern recognition that learns from examples on-line, generalizes and optimizes its storage capacity by off-line sleeping. Remarkably, the sleeping mechanisms always significantly reduce (up to ≈ 90%) the dataset size required to correctly generalize, further, there are memory loads that are prohibitive to Hebbian networks without sleeping (no matter the size and quality of the provided examples), but that are easily handled by the present "rested" neural networks.