mlsanityData to 7 Oct 2026

Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks

arXiv 2306.17630 ↗First posted 30 Jun 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Enhancing the generalisation abilities of neural networks (NNs) through integrating noise such as MixUp or Dropout during training has emerged as a powerful and adaptable technique. Despite the proven efficacy of noise in NN training, there is no consensus regarding which noise sources, types and placements yield maximal benefits in generalisation and confidence calibration. This study thoroughly explores diverse noise modalities to evaluate their impacts on NN's generalisation and calibration under in-distribution or out-of-distribution settings, paired with experiments investigating the metric landscapes of the learnt representations across a spectrum of NN architectures, tasks, and datasets. Our study shows that AugMix and weak augmentation exhibit cross-task effectiveness in computer vision, emphasising the need to tailor noise to specific domains. Our findings emphasise the efficacy of combining noises and successful hyperparameter transfer within a single domain but the difficulties in transferring the benefits to other domains. Furthermore, the study underscores the complexity of simultaneously optimising for both generalisation and calibration, emphasising the need for practitioners to carefully consider noise combinations and hyperparameter tuning for optimal performance in specific tasks and datasets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

36 of 58 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
24 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  4. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  6. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  8. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  9. 2018Generalisation in humans and deep neural networkscited by 237
  10. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  11. 2014Qualitatively characterizing neural network optimization problemscited by 186
  12. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  13. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  14. 2017Dataset Augmentation in Feature Spacecited by 124
  15. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  16. 2017Regularization for Deep Learning: A Taxonomycited by 91
  17. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  18. 2022C-Mixup: Improving Generalization in Regressioncited by 57
  19. 2017Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimizationcited by 56
  20. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  21. 2020An Empirical Evaluation on Robustness and Uncertainty of Regularization Methodscited by 28
  22. 2014Analyzing noise in autoencoders and deep networkscited by 25
  23. 2019Transfer of Adversarial Robustness Between Perturbation Typescited by 25
  24. 2015On the energy landscape of deep networkscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2025Staying on the Manifold: Geometry-Aware Noise Injectioncited by 4
  2. 2023DifAugGAN: A Practical Diffusion-style Data Augmentation for GAN-based Single Image Super-resolutioncited by 1
  3. 2024Policy Gradient-Driven Noise Maskcited by 1
  4. 2023The Benefit of Noise-Injection for Dynamic Gray-Box Model Creationcited by 0