mlsanityData to 7 Oct 2026

Gradient descent with generalized Newton's method

arXiv 2407.02772 ↗First posted 3 Jul 2024cs.LG cs.CL cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

We propose the generalized Newton's method (GeN) -- a Hessian-informed approach that applies to any optimizer such as SGD and Adam, and covers the Newton-Raphson method as a sub-case. Our method automatically and dynamically selects the learning rate that accelerates the convergence, without the intensive tuning of the learning rate scheduler. In practice, our method is easily implementable, since it only requires additional forward passes with almost zero computational overhead (in terms of training time and memory cost), if the overhead is amortized over many iterations. We present extensive experiments on language and vision tasks (e.g. GPT and ResNet) to showcase that GeN optimizers match the state-of-the-art performance, which was achieved with carefully tuned learning rate schedulers.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 50 of its 81 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

50 of 81 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2017Mask R-CNNcited by 7,259
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  13. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  14. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  15. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  16. 2023The Falcon Series of Open Language Modelscited by 526
  17. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  18. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  19. 2018An Empirical Model of Large-Batch Trainingcited by 268
  20. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  21. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  22. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  23. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  24. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  25. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  26. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 110
  27. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  28. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  29. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  30. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 68
  31. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  32. 2016Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimizationcited by 37
  33. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  34. 2016Efficient Second Order Online Learning by Sketchingcited by 33
  35. 2020Sketched Newton-Raphsoncited by 13
  36. 2015A Variance Reduced Stochastic Newton Methodcited by 8
  37. 2023QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learningcited by 6
  38. 2023ADLER -- An efficient Hessian-based strategy for adaptive learning ratecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 4.

9 papers.

  1. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  2. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  3. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  4. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  5. 2025A Hessian-informed hyperparameter optimization for differential learning ratecited by 1
  6. 2025Adaptive parameter-efficient fine-tuning via Hessian-informed subset selectioncited by 1
  7. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  8. 2026ZENITH: Automated Gradient Norm Informed Stochastic Optimizationcited by 0
  9. 2026Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuningcited by 0