mlsanityData to 14 Oct 2026

On the benefits of non-linear weight updates

arXiv 2207.12505 ↗First posted 25 Jul 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent work has suggested that the generalisation performance of a DNN is related to the extent to which the Signal-to-Noise Ratio is optimised at each of the nodes. In contrast, Gradient Descent methods do not always lead to SNR-optimal weight configurations. One way to improve SNR performance is to suppress large weight updates and amplify small weight updates. Such balancing is already implicit in some common optimizers, but we propose an approach that makes this explicit. The method applies a non-linear function to gradients prior to making DNN parameter updates. We investigate the performance with such non-linear approaches. The result is an adaptation to existing optimizers that improves performance for many problem types.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

30 of 57 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2016Wide Residual Networkscited by 3,070
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2016An overview of gradient descent optimization algorithmscited by 1,033
  9. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  10. 2019On the Convergence of Adam and Beyondcited by 841
  11. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  12. 2017Exploring Generalization in Deep Learningcited by 600
18 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  3. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  4. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  5. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  6. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  7. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
  8. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  9. 2017Generalization in Deep Learningcited by 146
  10. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  11. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  12. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  13. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 44
  14. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  15. 2020Representation Based Complexity Measures for Predicting Generalization in Deep Learningcited by 29
  16. 2021Convolutional Neural Networks Demystified: A Matched Filtering Perspective Based Tutorialcited by 6
  17. 2020Think Global, Act Local: Relating DNN generalisation and node-level SNRcited by 1
  18. 2020Using noise resilience for ranking generalization of deep neural networkscited by 1

Built on by

Not cited yet.