Perturbated Gradients Updating within Unit Space for Deep Learning
Not on the map. Why?
Abstract (arXiv)
In deep learning, optimization plays a vital role. By focusing on image classification, this work investigates the pros and cons of the widely used optimizers, and proposes a new optimizer: Perturbated Unit Gradient Descent (PUGD) algorithm with extending normalized gradient operation in tensor within perturbation to update in unit space. Via a set of experiments and analyses, we show that PUGD is locally bounded updating, which means the updating from time to time is controlled. On the other hand, PUGD can push models to a flat minimum, where the error remains approximately constant, not only because of the nature of avoiding stationary points in gradient normalization but also by scanning sharpness in the unit ball. From a series of rigorous experiments, PUGD helps models to gain a state-of-the-art Top-1 accuracy in Tiny ImageNet and competitive performances in CIFAR- {10, 100}. We open-source our code at link: https://github.com/hanktseng131415go/PUGD.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 1 Oct 2021.
Builds on
30 of 40 references.
References · 30
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2019A Comprehensive Survey on Transfer Learningcited by 1,154
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2017Exploring Generalization in Deep Learningcited by 595
- 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
18 more
- 2019Fantastic Generalization Measures and Where to Find Themcited by 507
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2016Deep Learning without Poor Local Minimacited by 233
- 2020Directional convergence and alignment in deep learningcited by 172
- 2019A Primer on PAC-Bayesian Learningcited by 130
- 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
- 2014Deep learning with Elastic Averaging SGDcited by 90
- 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
- 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
- 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
- 2019Generalization bounds for deep convolutional neural networkscited by 58
- 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
- 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 40
- 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
- 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
- 2021UPANets: Learning from the Universal Pixel Attention Networkscited by 7
Built on by
Citing papers per year: 2023: 3.
3 papers.