# Analyzing noise in autoencoders and deep networks

arXiv: https://arxiv.org/abs/1406.1831 · HTML: https://mlsanity.com/paper/1406.1831/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2014.

## Built on by (most cited first, 25 of 25)

- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Learning Smooth Neural Functions via Lipschitz Regularization](https://mlsanity.com/paper/2202.08345/) (2022)
- [Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate](https://mlsanity.com/paper/2402.02769/) (2024)
- [Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation](https://mlsanity.com/paper/2011.06719/) (2020)
- [NoiseGrad: Enhancing Explanations by Introducing Stochasticity to Model Weights](https://mlsanity.com/paper/2106.10185/) (2021)
- [On the Inherent Regularization Effects of Noise Injection During Training](https://mlsanity.com/paper/2102.07379/) (2021)
- [Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections](https://mlsanity.com/paper/2102.07006/) (2021)
- [GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs](https://mlsanity.com/paper/2408.15300/) (2024)
- [Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustness](https://mlsanity.com/paper/2310.18626/) (2023)
- [Knowledge Sharing via Domain Adaptation in Customs Fraud Detection](https://mlsanity.com/paper/2201.06759/) (2022)
- [Uncertainty Injection: A Deep Learning Method for Robust Optimization](https://mlsanity.com/paper/2302.12304/) (2023)
- [Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks](https://mlsanity.com/paper/2306.17630/) (2023)
- [Reinforcement Learning Platform for Adversarial Black-box Attacks with Custom Distortion Filters](https://mlsanity.com/paper/2501.14122/) (2025)
- [Insights into Lunar Mineralogy: An Unsupervised Approach for Clustering of the Moon Mineral Mapper (M3) spectral data](https://mlsanity.com/paper/2411.03186/) (2024)
- [Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automunge](https://mlsanity.com/paper/2202.09248/) (2022)
- [Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization](https://mlsanity.com/paper/2411.00383/) (2024)
- [Disturbing Target Values for Neural Network Regularization](https://mlsanity.com/paper/2110.05003/) (2021)
- [Stochastic resonance neurons in artificial neural networks](https://mlsanity.com/paper/2205.10122/) (2022)
- [Hyper-Reduced Autoencoders for Efficient and Accurate Nonlinear Model Reductions](https://mlsanity.com/paper/2303.09630/) (2023)
- [Layer-wise Regularized Dropout for Neural Language Models](https://mlsanity.com/paper/2402.16361/) (2024)
- [Beyond Trend Following: Deep Learning for Market Trend Prediction](https://mlsanity.com/paper/2407.13685/) (2024)
- [Harnessing the Power of Noise: A Survey of Techniques and Applications](https://mlsanity.com/paper/2410.06348/) (2024)
- [Limited-Resource Adapters Are Regularizers, Not Linguists](https://mlsanity.com/paper/2505.24525/) (2025)
- [Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisation](https://mlsanity.com/paper/2512.13707/) (2025)
- [Information-Regularized Attention for Visual-Centric Reasoning](https://mlsanity.com/paper/2607.00434/) (2026)

## Abstract (arXiv)

Autoencoders have emerged as a useful framework for unsupervised learning of internal representations, and a wide variety of apparently conceptually disparate regularization techniques have been proposed to generate useful features. Here we extend existing denoising autoencoders to additionally inject noise before the nonlinearity, and at the hidden unit activations. We show that a wide variety of previous methods, including denoising, contractive, and sparse autoencoders, as well as dropout can be interpreted using this framework. This noise injection framework reaps practical benefits by providing a unified strategy to develop new internal representations by designing the nature of the injected noise. We show that noisy autoencoders outperform denoising autoencoders at the very task of denoising, and are competitive with other single-layer techniques on MNIST, and CIFAR-10. We also show that types of noise other than dropout improve performance in a deep network through sparsifying, decorrelating, and spreading information across representations.