# Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization

arXiv: https://arxiv.org/abs/1710.05179 · HTML: https://mlsanity.com/paper/1710.05179/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2017.

## Built on by (most cited first, 30 of 56)

- [Quantum Vision Transformers](https://mlsanity.com/paper/2209.08167/) (2022)
- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [A Path to Simpler Models Starts With Noise](https://mlsanity.com/paper/2310.19726/) (2023)
- [Cross-View Cross-Scene Multi-View Crowd Counting](https://mlsanity.com/paper/2205.01551/) (2022)
- [On The Impact of Machine Learning Randomness on Group Fairness](https://mlsanity.com/paper/2307.04138/) (2023)
- [Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networks](https://mlsanity.com/paper/2305.16044/) (2023)
- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network](https://mlsanity.com/paper/2210.04349/) (2022)
- [A Kernel-Expanded Stochastic Neural Network](https://mlsanity.com/paper/2201.05319/) (2022)
- [ApproxTrain: Fast Simulation of Approximate Multipliers for DNN Training and Inference](https://mlsanity.com/paper/2209.04161/) (2022)
- [AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator](https://mlsanity.com/paper/2111.06503/) (2021)
- [The stochastic digital human is now enrolling for in silico imaging trials -- Methods and tools for generating digital cohorts](https://mlsanity.com/paper/2301.08719/) (2023)
- [Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics](https://mlsanity.com/paper/2309.04522/) (2023)
- [Are Ensembles Getting Better all the Time?](https://mlsanity.com/paper/2311.17885/) (2023)
- [Phase-driven Domain Generalizable Learning for Nonstationary Time Series](https://mlsanity.com/paper/2402.05960/) (2024)
- [SoK: A Review of Differentially Private Linear Models For High-Dimensional Data](https://mlsanity.com/paper/2404.01141/) (2024)
- [Supervision Adaptation Balancing In-distribution Generalization and Out-of-distribution Detection](https://mlsanity.com/paper/2206.09380/) (2022)
- [Training neural networks with structured noise improves classification and generalization](https://mlsanity.com/paper/2302.13417/) (2023)
- [Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks](https://mlsanity.com/paper/2306.17630/) (2023)
- [Syn2Real Domain Generalization for Underwater Mine-like Object Detection Using Side-Scan Sonar](https://mlsanity.com/paper/2410.12953/) (2024)
- [Gradient Extrapolation for Debiased Representation Learning](https://mlsanity.com/paper/2503.13236/) (2025)
- [Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models](https://mlsanity.com/paper/2506.07077/) (2025)
- [Spectral methods: crucial for machine learning, natural for quantum computers?](https://mlsanity.com/paper/2603.24654/) (2026)
- [Dynamic Cross Attention for Audio-Visual Person Verification](https://mlsanity.com/paper/2403.04661/) (2024)
- [Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.19554/) (2024)
- [A Unified Deep Semantic Expansion Framework for Domain-Generalized Person Re-identification](https://mlsanity.com/paper/2410.08456/) (2024)
- [Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML](https://mlsanity.com/paper/2411.11101/) (2024)
- [Towards a Robust Differentiable Architecture Search under Label Noise](https://mlsanity.com/paper/2110.12197/) (2021)
- [Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07372/) (2022)

## Abstract (arXiv)

Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.