mlsanityData to 14 Oct 2026

Revisiting Random Weight Perturbation for Efficiently Improving Generalization

arXiv 2404.00357 ↗First posted 30 Mar 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Improving the generalization ability of modern deep neural networks (DNNs) is a fundamental challenge in machine learning. Two branches of methods have been proposed to seek flat minima and improve generalization: one led by sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss through adversarial weight perturbation (AWP), and the other minimizes the expected Bayes objective with random weight perturbation (RWP). While RWP offers advantages in computation and is closely linked to AWP on a mathematical basis, its empirical performance has consistently lagged behind that of AWP. In this paper, we revisit the use of RWP for improving generalization and propose improvements from two perspectives: i) the trade-off between generalization and convergence and ii) the random perturbation generation. Through extensive experimental evaluations, we demonstrate that our enhanced RWP methods achieve greater efficiency in enhancing generalization, particularly in large-scale problems, while also offering comparable or even superior performance to SAM. The code is released at https://github.com/nblt/mARWP.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

41 of 54 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2016Wide Residual Networkscited by 3,070
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
29 more
  1. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  12. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  13. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  14. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  15. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  16. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  17. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  18. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  19. 2022Sharpness-Aware Training for Freecited by 95
  20. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  21. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  22. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  23. 2024Friendly Sharpness-Aware Minimizationcited by 53
  24. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  25. 2022SAM as an Optimal Relaxation of Bayescited by 41
  26. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  27. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  28. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  29. 2018SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learningcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 3.

13 papers.

Sort:

  1. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  2. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 10
  3. 2024Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategycited by 3
  4. 2025Accelerating Learned Image Compression Through Modeling Neural Training Dynamicscited by 1
  5. 2025Do Flat Minima Improve Sparse Novel View Synthesis?cited by 1
  6. 2024Revisiting Video Quality Assessment from the Perspective of Generalizationcited by 0
  7. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  8. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  9. 2025Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapescited by 0
  10. 2025Uncertainty-Aware Subset Selection for Robust Visual Explainability under Distribution Shiftscited by 0
  11. 2026Decentralized SGD with Controlled Disagreement Finds Flatter Minimacited by 0
  12. 2026Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbationscited by 0
1 more
  1. 2026Improving Sparse-View 3DGS Generalization via Flat Minima Optimizationcited by 0