mlsanityData to 7 Oct 2026

On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity

arXiv 2205.12642 ↗First posted 25 May 2022stat.ML cs.LG stat.COcited by 4

Not on the map. Why?

Abstract (arXiv)

Most complex machine learning and modelling techniques are prone to over-fitting and may subsequently generalise poorly to future data. Artificial neural networks are no different in this regard and, despite having a level of implicit regularisation when trained with gradient descent, often require the aid of explicit regularisers. We introduce a new framework, Model Gradient Similarity (MGS), that (1) serves as a metric of regularisation, which can be used to monitor neural network training, (2) adds insight into how explicit regularisers, while derived from widely different principles, operate via the same mechanism underneath by increasing MGS, and (3) provides the basis for a new regularisation scheme which exhibits excellent performance, especially in challenging settings such as high levels of label noise or limited sample sizes.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

19 of 32 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  9. 2020Implicit Gradient Regularizationcited by 145
  10. 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
  11. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  12. 2019Robust Learning with Jacobian Regularizationcited by 127
7 more
  1. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  2. 2021Input Similarity from the Neural Network Perspectivecited by 78
  3. 2019The Local Elasticity of Neural Networkscited by 37
  4. 2020A Study of Gradient Variance in Deep Learningcited by 27
  5. 2022On the Generalization Mystery in Deep Learningcited by 25
  6. 2019Inspecting adversarial examples using the Fisher informationcited by 7
  7. 2014An eigenanalysis of data centering in machine learningcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 0; 2025: 1.

4 papers.

  1. 2023Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray imagescited by 23
  2. 2023Low-Dimensional Gradient Helps Out-of-Distribution Detectioncited by 6
  3. 2025NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspectivecited by 2
  4. 2023Controlled Descent Trainingcited by 0