# Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models

arXiv: https://arxiv.org/abs/2508.21106 · HTML: https://mlsanity.com/paper/2508.21106/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 7 of 7)

- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descent](https://mlsanity.com/paper/1908.07861/) (2019)
- [Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks](https://mlsanity.com/paper/1611.02345/) (2016)
- [Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad](https://mlsanity.com/paper/2403.02648/) (2024)
- [Scalable Adaptive Stochastic Optimization Using Random Projections](https://mlsanity.com/paper/1611.06652/) (2016)
- [NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer](https://mlsanity.com/paper/2209.14937/) (2022)

## Built on by (most cited first, 1 of 1)

- [DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning](https://mlsanity.com/paper/2511.06477/) (2025)

## Abstract (arXiv)

Adaptive gradient methods like Adagrad and its variants are widespread in large-scale optimization. However, their use of diagonal preconditioning matrices limits the ability to capture parameter correlations. Full-matrix adaptive methods, approximating the exact Hessian, can model these correlations and may enable faster convergence. At the same time, their computational and memory costs are often prohibitive for large-scale models. To address this limitation, we propose AdaGram, an optimizer that enables efficient full-matrix adaptive gradient updates. To reduce memory and computational overhead, we utilize fast symmetric factorization for computing the preconditioned update direction at each iteration. Additionally, we maintain the low-rank structure of a preconditioner along the optimization trajectory using matrix integrator methods. Numerical experiments on standard machine learning tasks show that AdaGram converges faster or matches the performance of diagonal adaptive optimizers when using rank five and smaller rank approximations. This demonstrates AdaGram's potential as a scalable solution for adaptive optimization in large models.