# Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks

arXiv: https://arxiv.org/abs/2404.03992 · HTML: https://mlsanity.com/paper/2404.03992/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks](https://mlsanity.com/paper/1907.04595/) (2019)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [Stochastic Configuration Networks: Fundamentals and Algorithms](https://mlsanity.com/paper/1702.03180/) (2017)
- [Random Vector Functional Link Neural Network based Ensemble Deep Learning](https://mlsanity.com/paper/1907.00350/) (2019)
- [An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise](https://mlsanity.com/paper/1902.08234/) (2019)
- [Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm](https://mlsanity.com/paper/2006.12703/) (2020)
- [General stochastic separation theorems with optimal bounds](https://mlsanity.com/paper/2010.05241/) (2020)
- [Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection](https://mlsanity.com/paper/2111.09074/) (2021)

## Built on by (most cited first, 2 of 2)

- [DropKAN: Regularizing KANs by masking post-activations](https://mlsanity.com/paper/2407.13044/) (2024)
- [Regularisation in neural networks: a survey and empirical analysis of approaches](https://mlsanity.com/paper/2601.23131/) (2026)

## Abstract (arXiv)

This paper investigates how various randomization techniques impact Deep Neural Networks (DNNs). Randomization, like weight noise and dropout, aids in reducing overfitting and enhancing generalization, but their interactions are poorly understood. The study categorizes randomness techniques into four types and proposes new methods: adding noise to the loss function and random masking of gradient updates. Using Particle Swarm Optimizer (PSO) for hyperparameter optimization, it explores optimal configurations across MNIST, FASHION-MNIST, CIFAR10, and CIFAR100 datasets. Over 30,000 configurations are evaluated, revealing data augmentation and weight initialization randomness as main performance contributors. Correlation analysis shows different optimizers prefer distinct randomization types. The complete implementation and dataset are available on GitHub.