# Noisy Activation Functions

arXiv: https://arxiv.org/abs/1603.00391 · HTML: https://mlsanity.com/paper/1603.00391/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2016.

## Built on by (most cited first, 30 of 53)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [An Empirical Study of Implicit Regularization in Deep Offline RL](https://mlsanity.com/paper/2207.02099/) (2022)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [Learning Long Sequences in Spiking Neural Networks](https://mlsanity.com/paper/2401.00955/) (2023)
- [On the Inherent Regularization Effects of Noise Injection During Training](https://mlsanity.com/paper/2102.07379/) (2021)
- [NoisyMix: Boosting Model Robustness to Common Corruptions](https://mlsanity.com/paper/2202.01263/) (2022)
- [Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network](https://mlsanity.com/paper/2210.04349/) (2022)
- [Dynamic Flows on Curved Space Generated by Labeled Data](https://mlsanity.com/paper/2302.00061/) (2023)
- [A Kernel-Expanded Stochastic Neural Network](https://mlsanity.com/paper/2201.05319/) (2022)
- [TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation](https://mlsanity.com/paper/2307.14611/) (2023)
- [Noise Optimization for Artificial Neural Networks](https://mlsanity.com/paper/2102.04450/) (2021)
- [Over-the-Air Edge Inference via End-to-End Metasurfaces-Integrated Artificial Neural Networks](https://mlsanity.com/paper/2504.00233/) (2025)
- [Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks](https://mlsanity.com/paper/2301.09245/) (2023)
- [LiteLSTM Architecture for Deep Recurrent Neural Networks](https://mlsanity.com/paper/2201.11624/) (2022)
- [Self-scalable Tanh (Stan): Faster Convergence and Better Generalization in Physics-informed Neural Networks](https://mlsanity.com/paper/2204.12589/) (2022)
- [Uncertainty Injection: A Deep Learning Method for Robust Optimization](https://mlsanity.com/paper/2302.12304/) (2023)
- [Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients](https://mlsanity.com/paper/2503.09206/) (2025)
- [BubbleOKAN: A Physics-Informed Interpretable Neural Operator for High-Frequency Bubble Dynamics](https://mlsanity.com/paper/2508.03965/) (2025)
- [Why KDAC? A general activation function for knowledge discovery](https://mlsanity.com/paper/2111.13858/) (2021)
- [Neural Networks with A La Carte Selection of Activation Functions](https://mlsanity.com/paper/2206.12166/) (2022)
- [Instability in clinical risk stratification models using deep learning](https://mlsanity.com/paper/2211.10828/) (2022)
- [PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusion](https://mlsanity.com/paper/2305.15835/) (2023)
- [Real-Time Optimal Trajectory Planning for Autonomous Vehicles and Lap Time Simulation Using Machine Learning](https://mlsanity.com/paper/2102.02315/) (2021)
- [Towards a Robust Differentiable Architecture Search under Label Noise](https://mlsanity.com/paper/2110.12197/) (2021)
- [Langevin algorithms for Markovian Neural Networks and Deep Stochastic control](https://mlsanity.com/paper/2212.12018/) (2022)
- [Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks](https://mlsanity.com/paper/2404.03992/) (2024)
- [Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations](https://mlsanity.com/paper/2503.16183/) (2025)

## Abstract (arXiv)

Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent toexplore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.