mlsanityData to 14 Oct 2026

SING: A Plug-and-Play DNN Learning Technique

arXiv 2305.15997 ↗First posted 25 May 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of the gradients fed to Adam(W) without introducing additional hyper-parameters. We support the effectiveness and practicality of the proposed approach by showing improved results on a wide range of architectures, problems (such as image classification, depth estimation, and natural language processing), and in combination with other optimizers. We provide a theoretical analysis of the convergence of the method, and we show that by virtue of the standardization, SING can escape local minima narrower than a threshold that is inversely proportional to the network's depth.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

42 of 60 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2016Layer Normalizationcited by 5,669
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
30 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2020On Layer Normalization in the Transformer Architecturecited by 922
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  5. 2021Going deeper with Image Transformerscited by 764
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  10. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  11. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  12. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  13. 2019Understanding and Improving Layer Normalizationcited by 217
  14. 2019Fixing the train-test resolution discrepancycited by 209
  15. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  16. 2014Qualitatively characterizing neural network optimization problemscited by 188
  17. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  18. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  19. 2020Momentum Improves Normalized SGDcited by 139
  20. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  21. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  22. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
  23. 2018The loss landscape of overparameterized neural networkscited by 39
  24. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  25. 2017Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Pointscited by 29
  26. 2020Gradient Amplification: An efficient way to train deep neural networkscited by 15
  27. 2019An Adaptive and Momental Bound Method for Stochastic Learningcited by 12
  28. 2020EAdam Optimizer: How ε Impact Adamcited by 10
  29. 2022AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNscited by 5
  30. 2022Investigating Neural Architectures by Synthetic Dataset Designcited by 3

Built on by

Not cited yet.