# Beyond First-Order: Training LLMs with Stochastic Conjugate Subgradients and AdamW

arXiv: https://arxiv.org/abs/2507.01241 · HTML: https://mlsanity.com/paper/2507.01241/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving](https://mlsanity.com/paper/2505.17685/) (2025)
- [Orthogonal Random Features](https://mlsanity.com/paper/1610.09072/) (2016)
- [Optimization Methods for Supervised Machine Learning: From Linear Models to Deep Learning](https://mlsanity.com/paper/1706.10207/) (2017)
- [Stochastic Conjugate Gradient Algorithm with Variance Reduction](https://mlsanity.com/paper/1710.09979/) (2017)
- [The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines](https://mlsanity.com/paper/2407.21091/) (2024)
- [A Reliability Theory of Compromise Decisions for Large-Scale Stochastic Programs](https://mlsanity.com/paper/2405.10414/) (2024)
- [A Stochastic Conjugate Subgradient Algorithm for Two-stage Stochastic Programming](https://mlsanity.com/paper/2503.21053/) (2025)

## Built on by (most cited first, 2 of 2)

- [Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations](https://mlsanity.com/paper/2604.12968/) (2026)
- [The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models](https://mlsanity.com/paper/2510.23652/) (2025)

## Abstract (arXiv)

Stochastic gradient-based descent (SGD), have long been central to training large language models (LLMs). However, their effectiveness is increasingly being questioned, particularly in large-scale applications where empirical evidence suggests potential performance limitations. In response, this paper proposes a stochastic conjugate subgradient method together with adaptive sampling tailored specifically for training LLMs. The method not only achieves faster convergence per iteration but also demonstrates improved scalability compared to traditional SGD techniques. It leverages sample complexity analysis to adaptively choose the sample size, employs a stochastic conjugate subgradient approach to determine search directions and utilizing an AdamW-like algorithm to adaptively adjust step sizes. This approach preserves the key advantages of first-order methods while effectively addressing the nonconvexity and non-smoothness inherent in LLMs training. Additionally, we provide a detailed analysis of the advantage of the algorithm. Experimental results show that the proposed method not only maintains, but in many cases surpasses, the scalability of traditional SGD techniques, significantly enhancing both the speed and accuracy of the optimization process.