mlsanityData to 7 Oct 2026

Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning

arXiv 2202.03599 ↗First posted 8 Feb 2022cs.LG cs.AIcited by 122

Not on the map. Why?

Abstract (arXiv)

How to train deep neural networks (DNNs) to generalize well is a central concern in deep learning, especially for severely overparameterized networks nowadays. In this paper, we propose an effective method to improve the model generalization by additionally penalizing the gradient norm of loss function during optimization. We demonstrate that confining the gradient norm of loss function could help lead the optimizers towards finding flat minima. We leverage the first-order approximation to efficiently implement the corresponding gradient to fit well in the gradient descent framework. In our experiments, we confirm that when using our methods, generalization performance of various models could be improved on different datasets. Also, we show that the recent sharpness-aware minimization method (Foret et al., 2021) is a special, but not the best, case of our method, where the best case of our method could give new state-of-art performance on these tasks. Code is available at {https://github.com/zhaoyang-0204/gnp}.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

23 of 34 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2016Wide Residual Networkscited by 3,057
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2018Group Normalizationcited by 1,514
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
11 more
  1. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  2. 2017Exploring Generalization in Deep Learningcited by 595
  3. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  4. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  7. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  8. 2016Deep Pyramidal Residual Networkscited by 163
  9. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  10. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  11. 2018ShakeDrop Regularization for Deep Residual Learningcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 36; 2024: 34; 2025: 32; 2026: 12.

122 papers.

Sort:

By area:

  1. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  2. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  3. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  4. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  5. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  6. 2023Boosting Adversarial Transferability by Achieving Flat Local Maximacited by 48
  7. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  8. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  9. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  10. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  11. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  12. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
38 more
  1. 2023Small batch deep reinforcement learningcited by 35
  2. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  3. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  4. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29
  5. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  6. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  7. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  8. 2023On the Implicit Bias of Adamcited by 25
  9. 2023FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learningcited by 20
  10. 2024A Unified and General Framework for Continual Learningcited by 19
  11. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  12. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  13. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  14. 2022Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learningcited by 14
  15. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  16. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  17. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  18. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  19. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  20. 2024Domain-Inspired Sharpness-Aware Minimization Under Domain Shiftscited by 12
  21. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  22. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  23. 2025Self-Destructive Language Modelcited by 10
  24. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  25. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  26. 2025DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Normcited by 9
  27. 2022Efficient Generalization Improvement Guided by Random Weight Perturbationcited by 8
  28. 2023Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Trainingcited by 8
  29. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  30. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  31. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  32. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  33. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  34. 2024TASAR: Transfer-based Attack on Skeletal Action Recognitioncited by 7
  35. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Modelscited by 7
  36. 2023Improving Differentiable Architecture Search via Self-Distillationcited by 6
  37. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  38. 2023DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learningcited by 6

50 most cited of 122 All 122 (JSON)