# Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis

arXiv: https://arxiv.org/abs/2510.11312 · HTML: https://mlsanity.com/paper/2510.11312/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2025.

## Built on by (most cited first, 2 of 2)

- [Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime](https://mlsanity.com/paper/2603.10485/) (2026)
- [Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives](https://mlsanity.com/paper/2605.11850/) (2026)

## Abstract (arXiv)

We study nonlinearly preconditioned gradient methods for smooth nonconvex optimization problems, focusing on sigmoid preconditioners that inherently perform a form of gradient clipping akin to the widely used gradient clipping technique. Building upon this idea, we introduce a novel heavy ball-type algorithm and provide convergence guarantees under a generalized smoothness condition that is less restrictive than traditional Lipschitz smoothness, thus covering a broader class of functions. Additionally, we develop a stochastic variant of the base method and study its convergence properties under different noise assumptions. We compare the proposed algorithms with baseline methods on diverse tasks from machine learning including neural network training.