# Dreaming Learning

arXiv: https://arxiv.org/abs/2410.18156 · HTML: https://mlsanity.com/paper/2410.18156/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 13 references that are in the map.
- First posted on arXiv 23 Oct 2024.

## Builds on (most cited first, 5 of 5)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [The Overfitted Brain: Dreams evolved to assist generalization](https://mlsanity.com/paper/2007.09560/) (2020)
- [The dynamics of correlated novelties](https://mlsanity.com/paper/1310.1953/) (2013)
- [Natural Language Statistical Features of LSTM-generated Texts](https://mlsanity.com/paper/1804.04087/) (2018)

## Built on by (most cited first, 1 of 1)

- [Lyapunov Learning at the Onset of Chaos](https://mlsanity.com/paper/2506.12810/) (2025)

## Abstract (arXiv)

Incorporating novelties into deep learning systems remains a challenging problem. Introducing new information to a machine learning system can interfere with previously stored data and potentially alter the global model paradigm, especially when dealing with non-stationary sources. In such cases, traditional approaches based on validation error minimization offer limited advantages. To address this, we propose a training algorithm inspired by Stuart Kauffman's notion of the Adjacent Possible. This novel training methodology explores new data spaces during the learning phase. It predisposes the neural network to smoothly accept and integrate data sequences with different statistical characteristics than expected. The maximum distance compatible with such inclusion depends on a specific parameter: the sampling temperature used in the explorative phase of the present method. This algorithm, called Dreaming Learning, anticipates potential regime shifts over time, enhancing the neural network's responsiveness to non-stationary events that alter statistical properties. To assess the advantages of this approach, we apply this methodology to unexpected statistical changes in Markov chains and non-stationary dynamics in textual sequences. We demonstrated its ability to improve the auto-correlation of generated textual sequences by ~ 29% and enhance the velocity of loss convergence by ~ 100% in the case of a paradigm shift in Markov chains.