# Learning Dynamics of Linear Denoising Autoencoders

arXiv: https://arxiv.org/abs/1806.05413 · HTML: https://mlsanity.com/paper/1806.05413/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 16 of 16)

- [MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms](https://mlsanity.com/paper/2111.03187/) (2021)
- [The dynamics of representation learning in shallow, non-linear autoencoders](https://mlsanity.com/paper/2201.02115/) (2022)
- [High-dimensional Asymptotics of Denoising Autoencoders](https://mlsanity.com/paper/2305.11041/) (2023)
- [Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling](https://mlsanity.com/paper/2502.05743/) (2025)
- [When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning](https://mlsanity.com/paper/2406.17718/) (2024)
- [A solvable model of learning generative diffusion: theory and insights](https://mlsanity.com/paper/2501.03937/) (2025)
- [Non-Parametric Representation Learning with Kernels](https://mlsanity.com/paper/2309.02028/) (2023)
- [Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequencies](https://mlsanity.com/paper/2303.02034/) (2023)
- [Underwater enhancement based on a self-learning strategy and attention mechanism for high-intensity regions](https://mlsanity.com/paper/2208.03319/) (2022)
- [Representation Learning Dynamics of Self-Supervised Models](https://mlsanity.com/paper/2309.02011/) (2023)
- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [Improved Representation Learning Through Tensorized Autoencoders](https://mlsanity.com/paper/2212.01046/) (2022)
- [Redundancy in Deep Linear Neural Networks](https://mlsanity.com/paper/2206.04490/) (2022)
- [Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders](https://mlsanity.com/paper/2505.24668/) (2025)
- [A prism hierarchy of learning regimes in large linear autoencoders](https://mlsanity.com/paper/2606.05335/) (2026)
- [Diagonal Attenuation: A Finite-Sample Correction for PCA](https://mlsanity.com/paper/2609.05796/) (2026)

## Abstract (arXiv)

Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs.