mlsanityData to 14 Oct 2026

An Adaptive Method Stabilizing Activations for Enhanced Generalization

arXiv 2506.08353 ↗First posted 10 Jun 2025cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce AdaAct, a novel optimization algorithm that adjusts learning rates according to activation variance. Our method enhances the stability of neuron outputs by incorporating neuron-wise adaptivity during the training process, which subsequently leads to better generalization -- a complementary approach to conventional activation regularization methods. Experimental results demonstrate AdaAct's competitive performance across standard image classification benchmarks. We evaluate AdaAct on CIFAR and ImageNet, comparing it with other state-of-the-art methods. Importantly, AdaAct effectively bridges the gap between the convergence speed of Adam and the strong generalization capabilities of SGD, all while maintaining competitive execution times. Code is available at https://github.com/hseung88/adaact.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2016Layer Normalizationcited by 5,669
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
26 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2016Deep Networks with Stochastic Depthcited by 979
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  5. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  8. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  9. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  10. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  11. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  12. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  13. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  14. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  15. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  16. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  17. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  18. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  19. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  20. 2020Adam^+: A Stochastic Method with Adaptive Variance Reductioncited by 18
  21. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  22. 2015Regularizing RNNs by Stabilizing Activationscited by 6
  23. 2017Revisiting Activation Regularization for Language RNNscited by 6
  24. 2018Regularizing by the Variance of the Activations' Sample-Variancescited by 5
  25. 2020Regularizing activations in neural networks via distribution matching with the Wasserstein metriccited by 4
  26. 2021Neuron with Steady Response Leads to Better Generalizationcited by 4

Built on by

Not cited yet.