# Don't Be So Positive: Negative Step Sizes in Second-Order Methods

arXiv: https://arxiv.org/abs/2411.11224 · HTML: https://mlsanity.com/paper/2411.11224/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Adaptive Gradient Descent without Descent](https://mlsanity.com/paper/1910.09529/) (2019)
- [On the saddle point problem for non-convex optimization](https://mlsanity.com/paper/1405.4604/) (2014)
- [A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Points](https://mlsanity.com/paper/1707.08028/) (2017)
- [Cubic Regularization is the Key! The First Accelerated Quasi-Newton Method with a Global Convergence Rate of O(k-2) for Convex Functions](https://mlsanity.com/paper/2302.04987/) (2023)

## Built on by (most cited first, 1 of 1)

- [Negative Stepsizes Make Gradient-Descent-Ascent Converge](https://mlsanity.com/paper/2505.01423/) (2025)

## Abstract (arXiv)

The value of second-order methods lies in the use of curvature information. Yet, this information is costly to extract and once obtained, valuable negative curvature information is often discarded so that the method is globally convergent. This limits the effectiveness of second-order methods in modern machine learning. In this paper, we show that second-order and second-order-like methods are promising optimizers for neural networks provided that we add one ingredient: negative step sizes. We show that under very general conditions, methods that produce ascent directions are globally convergent when combined with a Wolfe line search that allows both positive and negative step sizes. We experimentally demonstrate that using negative step sizes is often more effective than common Hessian modification methods.