# Scalable Second Order Optimization for Deep Learning

arXiv: https://arxiv.org/abs/2002.09018 · HTML: https://mlsanity.com/paper/2002.09018/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 21 of 21)

- [Real-time Neural Radiance Caching for Path Tracing](https://mlsanity.com/paper/2106.12372/) (2021)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Generalizable Approach to Learning Optimizers](https://mlsanity.com/paper/2106.00958/) (2021)
- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization](https://mlsanity.com/paper/2411.10696/) (2024)
- [Learning Physics Informed Neural ODEs With Partial Measurements](https://mlsanity.com/paper/2412.08681/) (2024)
- [Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics](https://mlsanity.com/paper/2303.05101/) (2023)
- [PETScML: Second-order solvers for training regression problems in Scientific Machine Learning](https://mlsanity.com/paper/2403.12188/) (2024)
- [Hessian-Free Second-Order Adversarial Examples for Adversarial Learning](https://mlsanity.com/paper/2207.01396/) (2022)
- [Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers](https://mlsanity.com/paper/2605.09176/) (2026)
- [Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss](https://mlsanity.com/paper/2606.06418/) (2026)
- [Optimizer Memory Schedules for Outscaling the Overtraining Axis](https://mlsanity.com/paper/2609.04577/) (2026)
- [Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size](https://mlsanity.com/paper/2112.14872/) (2021)
- [FUSE: First-Order and Second-Order Unified SynthEsis in Stochastic Optimization](https://mlsanity.com/paper/2503.04204/) (2025)
- [Fundamentals of Regression](https://mlsanity.com/paper/2512.01920/) (2025)
- [Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression](https://mlsanity.com/paper/2608.24070/) (2026)
- [How Bregman Divergences Shape Shampoo](https://mlsanity.com/paper/2610.08534/) (2026)

## Abstract (arXiv)

Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent. Second-order optimization methods, that involve second derivatives and/or second order statistics of the data, are far less prevalent despite strong theoretical properties, due to their prohibitive computation, memory and communication costs. In an attempt to bridge this gap between theoretical and practical optimization, we present a scalable implementation of a second-order preconditioned method (concretely, a variant of full-matrix Adagrad), that along with several critical algorithmic and numerical improvements, provides significant convergence and wall-clock time improvements compared to conventional first-order methods on state-of-the-art deep models. Our novel design effectively utilizes the prevalent heterogeneous hardware architecture for training deep models, consisting of a multicore CPU coupled with multiple accelerator units. We demonstrate superior performance compared to state-of-the-art on very large learning tasks such as machine translation with Transformers, language modeling with BERT, click-through rate prediction on Criteo, and image classification on ImageNet with ResNet-50.