mlsanityData to 14 Oct 2026

A second-order-like optimizer with adaptive gradient scaling for deep learning

arXiv 2410.05871 ↗First posted 8 Oct 2024cs.LG cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After giving geometrical insights, we evaluate INNAprop on CIFAR-10, Food101, and ImageNet with ResNets, VGG, DenseNet, and ViT, and on GPT-2 (OpenWebText) train from scratch and with LoRA fine-tuning (E2E). INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at urlhttps://github.com/innaprop/innaprop.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 77 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

49 of 77 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  12. 2016An overview of gradient descent optimization algorithmscited by 1,033
37 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  5. 2019Green AIcited by 627
  6. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  7. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  10. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  11. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  12. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  13. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  14. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  15. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  16. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 169
  17. 2024The Road Less Scheduledcited by 163
  18. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  19. 2018Stochastic subgradient method converges on tame functionscited by 124
  20. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 113
  21. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  22. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
  23. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
  24. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  25. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  26. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  27. 2024Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AIcited by 35
  28. 2019First-order optimization algorithms via inertial systems with Hessian driven dampingcited by 22
  29. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  30. 2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 15
  31. 2019An Inertial Newton Algorithm for Deep Learningcited by 14
  32. 2016Fast convex optimization via inertial dynamics with Hessian driven dampingcited by 13
  33. 2017Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3cited by 9
  34. 2018Optimal convergence rates for Nesterov accelerationcited by 8
  35. 2019First order optimization methods based on Hessian-driven Nesterov accelerated gradient flowcited by 6
  36. 2023Continuous Newton-like Methods featuring Inertia and Variable Masscited by 2
  37. 2024Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligencecited by 2

Built on by

Not cited yet.