mlsanityData to 14 Oct 2026

Improving Generalization and Convergence by Enhancing Implicit Regularization

arXiv 2405.20763 ↗First posted 31 May 2024cs.LG math.OC stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with em generic base optimizers without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). Surprisingly, IRE also achieves a 2× em speed-up compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in Sharpness-aware Minimization (SAM).

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 83 of its 108 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

83 of 108 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2016Wide Residual Networkscited by 3,070
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
38 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2019Root Mean Square Layer Normalizationcited by 1,480
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  13. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  14. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  15. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  16. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  17. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  18. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  19. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  20. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  21. 2017Three Factors Influencing Minima in SGDcited by 273
  22. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  23. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  24. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  25. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  26. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  27. 2020Directional convergence and alignment in deep learningcited by 178
  28. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  29. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  30. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  31. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  32. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  33. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  34. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  35. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  36. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  37. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  38. 2018Convergence of Gradient Descent on Separable Datacited by 110

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 7.

14 papers.

Sort:

By area:

  1. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  2. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
  3. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  4. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  5. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 8
  6. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  7. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 4
  8. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Modelscited by 4
  9. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  10. 2025BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Trainingcited by 3
  11. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  12. 2026Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Trainingcited by 2
2 more
  1. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
  2. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0