# Online Estimation with Rolling Validation: Adaptive Nonparametric Estimation with Streaming Data

arXiv: https://arxiv.org/abs/2310.12140 · HTML: https://mlsanity.com/paper/2310.12140/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)](https://mlsanity.com/paper/1306.2119/) (2013)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Model Selection Techniques -- An Overview](https://mlsanity.com/paper/1810.09583/) (2018)
- [Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses](https://mlsanity.com/paper/1907.01771/) (2019)
- [Cross-Validation with Confidence](https://mlsanity.com/paper/1703.07904/) (2017)
- [Online Estimation for Functional Data](https://mlsanity.com/paper/2111.02750/) (2021)
- [Risk-consistency of cross-validation with lasso-type procedures](https://mlsanity.com/paper/1308.0810/) (2013)
- [A Sieve Stochastic Gradient Descent Estimator for Online Nonparametric Regression in Sobolev ellipsoids](https://mlsanity.com/paper/2104.00846/) (2021)
- [Regression in Tensor Product Spaces by the Method of Sieves](https://mlsanity.com/paper/2206.02994/) (2022)
- [An Online Projection Estimator for Nonparametric Regression in Reproducing Kernel Hilbert Spaces](https://mlsanity.com/paper/2104.00780/) (2021)

## Built on by (most cited first, 1 of 1)

- [Stochastic Gradient Descent for Nonparametric Additive Regression](https://mlsanity.com/paper/2401.00691/) (2024)

## Abstract (arXiv)

Online nonparametric estimators are gaining popularity due to their efficient computation and competitive generalization abilities. An important example includes variants of stochastic gradient descent. These algorithms often take one sample point at a time and incrementally update the parameter estimate of interest. In this work, we consider model selection/hyperparameter tuning for such online algorithms. We propose a weighted rolling validation procedure, an online variant of leave-one-out cross-validation, that costs minimal extra computation for many typical stochastic gradient descent estimators and maintains their online nature. Similar to batch cross-validation, it can boost base estimators to achieve better heuristic performance and adaptive convergence rate. Our analysis is straightforward, relying mainly on some general statistical stability assumptions. The simulation study underscores the significance of diverging weights in practice and demonstrates its favorable sensitivity even when there is only a slim difference between candidate estimators.