mlsanityData to 14 Oct 2026

Regularizing Neural Networks via Adversarial Model Perturbation

arXiv 2010.04925 ↗First posted 10 Oct 2020cs.LG cs.AI stat.MLcited by 78

Not on the map. Why?

Abstract (arXiv)

Effective regularization techniques are highly desired in deep learning for alleviating overfitting and improving generalization. This work proposes a new regularization scheme, based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. This scheme is referred to as adversarial model perturbation (AMP), where instead of directly minimizing the empirical risk, an alternative "AMP loss" is minimized via SGD. Specifically, the AMP loss is obtained from the empirical risk by applying the "worst" norm-bounded perturbation on each point in the parameter space. Comparing with most existing regularization schemes, AMP has strong theoretical justifications, in that minimizing the AMP loss can be shown theoretically to favour flat local minima of the empirical risk. Extensive experiments on various modern deep architectures establish AMP as a new state of the art among regularization schemes. Our code is available at https://github.com/hiyouga/AMP-Regularizer.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 22; 2023: 16; 2024: 15; 2025: 9; 2026: 5.

78 papers.

Sort:

By area:

  1. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 415
  2. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  4. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  5. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  6. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  7. 2022Sharpness-Aware Training for Freecited by 95
  8. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  9. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  10. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  11. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  12. 2024Friendly Sharpness-Aware Minimizationcited by 53
38 more
  1. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  2. 2022SAM as an Optimal Relaxation of Bayescited by 41
  3. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  4. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  5. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29
  6. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  7. 2022Benchopt: Reproducible, efficient and collaborative optimization benchmarkscited by 27
  8. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  9. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  10. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  11. 2022Adversarial Weight Perturbation Improves Generalization in Graph Neural Networkscited by 21
  12. 2024Why is SAM Robust to Label Noise?cited by 21
  13. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  14. 2023Improving the Accuracy-Robustness Trade-Off of Classifiers via Adaptive Smoothingcited by 18
  15. 2023Loss Spike in Training Neural Networkscited by 16
  16. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  17. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  18. 2022Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learningcited by 14
  19. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 14
  20. 2021Learning to Learn Transferable Attackcited by 13
  21. 2022Improving Adversarial Robustness via Mutual Information Estimationcited by 13
  22. 2022GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalizationcited by 13
  23. 2022Paraphrase Identification with Deep Learning: A Review of Datasets and Methodscited by 13
  24. 2023How to escape sharp minima with random perturbationscited by 13
  25. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  26. 2022Continual Learning by Modeling Intra-Class Variationcited by 12
  27. 2025Sharpness-Aware Minimization: General Analysis and Improved Ratescited by 10
  28. 2021Understanding and Improvement of Adversarial Training for Network Embedding from an Optimization Perspectivecited by 9
  29. 2022Efficient Generalization Improvement Guided by Random Weight Perturbationcited by 8
  30. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  31. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  32. 2024DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-trainingcited by 7
  33. 2021Sharpness-Aware Minimization with Dynamic Reweightingcited by 6
  34. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  35. 2024Tilted Sharpness-Aware Minimizationcited by 6
  36. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  37. 2021Network insensitivity to parameter noise via adversarial regularizationcited by 5
  38. 2023Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiencycited by 5

50 most cited of 78 All 78 (JSON)