# Stochastic gradient descent methods for estimation with large data sets

arXiv: https://arxiv.org/abs/1509.06459 · HTML: https://mlsanity.com/paper/1509.06459/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2015.

## Built on by (most cited first, 2 of 2)

- [Scalable kernel balancing weights in a nationwide observational study of hospital profit status and heart attack outcomes](https://mlsanity.com/paper/2311.00568/) (2023)
- [High Dimensional Time Series Regression Models: Applications to Statistical Learning Methods](https://mlsanity.com/paper/2308.16192/) (2023)

## Abstract (arXiv)

We develop methods for parameter estimation in settings with large-scale data sets, where traditional methods are no longer tenable. Our methods rely on stochastic approximations, which are computationally efficient as they maintain one iterate as a parameter estimate, and successively update that iterate based on a single data point. When the update is based on a noisy gradient, the stochastic approximation is known as standard stochastic gradient descent, which has been fundamental in modern applications with large data sets. Additionally, our methods are numerically stable because they employ implicit updates of the iterates. Intuitively, an implicit update is a shrinked version of a standard one, where the shrinkage factor depends on the observed Fisher information at the corresponding data point. This shrinkage prevents numerical divergence of the iterates, which can be caused either by excess noise or outliers. Our sgd package in R offers the most extensive and robust implementation of stochastic gradient descent methods. We demonstrate that sgd dominates alternative software in runtime for several estimation problems with massive data sets. Our applications include the wide class of generalized linear models as well as M-estimation for robust regression.