# Learning cortical representations through perturbed and adversarial dreaming

arXiv: https://arxiv.org/abs/2109.04261 · HTML: https://mlsanity.com/paper/2109.04261/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 151 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2001.06937/) (2020)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [The Predictron: End-To-End Learning and Planning](https://mlsanity.com/paper/1612.08810/) (2016)
- [Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer](https://mlsanity.com/paper/1807.07543/) (2018)
- [IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis](https://mlsanity.com/paper/1807.06358/) (2018)
- [Towards an integration of deep learning and neuroscience](https://mlsanity.com/paper/1606.03813/) (2016)
- [Towards deep learning with segregated dendrites](https://mlsanity.com/paper/1610.00161/) (2016)
- [Reweighted Wake-Sleep](https://mlsanity.com/paper/1406.2751/) (2014)
- [It Takes (Only) Two: Adversarial Generator-Encoder Networks](https://mlsanity.com/paper/1704.02304/) (2017)
- [The Overfitted Brain: Dreams evolved to assist generalization](https://mlsanity.com/paper/2007.09560/) (2020)

## Built on by (most cited first, 10 of 10)

- [Is artificial consciousness achievable? Lessons from the human brain](https://mlsanity.com/paper/2405.04540/) (2024)
- [The developmental trajectory of object recognition robustness: children are like small adults but unlike big deep neural networks](https://mlsanity.com/paper/2205.10144/) (2022)
- [Training neural networks with structured noise improves classification and generalization](https://mlsanity.com/paper/2302.13417/) (2023)
- [Wake-Sleep Consolidated Learning](https://mlsanity.com/paper/2401.08623/) (2023)
- [Learning beyond sensations: how dreams organize neuronal representations](https://mlsanity.com/paper/2308.01830/) (2023)
- [Sufficient conditions for offline reactivation in recurrent neural networks](https://mlsanity.com/paper/2505.17003/) (2025)
- [GENESIS: A Generative Model of Episodic-Semantic Interaction](https://mlsanity.com/paper/2510.15828/) (2025)
- [Sleep-Based Homeostatic Regularization for Stabilizing Spike-Timing-Dependent Plasticity in Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2601.08447/) (2026)
- [Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay](https://mlsanity.com/paper/2602.18401/) (2026)
- [Dream2Learn: Structured Generative Dreaming for Continual Learning](https://mlsanity.com/paper/2603.01935/) (2026)

## Abstract (arXiv)

Humans and other animals learn to extract general concepts from sensory experience without extensive teaching. This ability is thought to be facilitated by offline states like sleep where previous experiences are systemically replayed. However, the characteristic creative nature of dreams suggests that learning semantic representations may go beyond merely replaying previous experiences. We support this hypothesis by implementing a cortical architecture inspired by generative adversarial networks (GANs). Learning in our model is organized across three different global brain states mimicking wakefulness, NREM and REM sleep, optimizing different, but complementary objective functions. We train the model on standard datasets of natural images and evaluate the quality of the learned representations. Our results suggest that generating new, virtual sensory inputs via adversarial dreaming during REM sleep is essential for extracting semantic concepts, while replaying episodic memories via perturbed dreaming during NREM sleep improves the robustness of latent representations. The model provides a new computational perspective on sleep states, memory replay and dreams and suggests a cortical implementation of GANs.