# Variance Reduced Training with Stratified Sampling for Forecasting Models

arXiv: https://arxiv.org/abs/2103.02062 · HTML: https://mlsanity.com/paper/2103.02062/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 65 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [SGD: General Analysis and Improved Rates](https://mlsanity.com/paper/1901.09401/) (2019)
- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [Lower Bounds for Finding Stationary Points I](https://mlsanity.com/paper/1710.11606/) (2017)
- [Online Batch Selection for Faster Training of Neural Networks](https://mlsanity.com/paper/1511.06343/) (2015)
- [Deep Factors for Forecasting](https://mlsanity.com/paper/1905.12417/) (2019)
- [Variance Reduction in SGD by Distributed Importance Sampling](https://mlsanity.com/paper/1511.06481/) (2015)
- [Principles and Algorithms for Forecasting Groups of Time Series: Locality and Globality](https://mlsanity.com/paper/2008.00444/) (2020)
- [Probabilistic Forecasting with Temporal Convolutional Neural Network](https://mlsanity.com/paper/1906.04397/) (2019)
- [Non-convex Finite-Sum Optimization Via SCSG Methods](https://mlsanity.com/paper/1706.09156/) (2017)
- [SGD without Replacement: Sharper Rates for General Smooth Convex Functions](https://mlsanity.com/paper/1903.01463/) (2019)
- [On the Ineffectiveness of Variance Reduced Optimization for Deep Learning](https://mlsanity.com/paper/1812.04529/) (2018)
- [GluonTS: Probabilistic Time Series Models in Python](https://mlsanity.com/paper/1906.05264/) (2019)
- [Importance Sampling for Minibatches](https://mlsanity.com/paper/1602.02283/) (2016)
- [FAE: A Fairness-Aware Ensemble Framework](https://mlsanity.com/paper/2002.00695/) (2020)
- [A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/1802.04477/) (2018)
- [A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent](https://mlsanity.com/paper/1709.06617/) (2017)
- [LSTM-MSNet: Leveraging Forecasts on Sets of Related Time Series with Multiple Seasonal Patterns](https://mlsanity.com/paper/1909.04293/) (2019)
- [Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling](https://mlsanity.com/paper/1405.3080/) (2014)
- [Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/2002.05359/) (2020)
- [Stop Wasting My Gradients: Practical SVRG](https://mlsanity.com/paper/1511.01942/) (2015)

## Built on by (most cited first, 14 of 14)

- [Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models](https://mlsanity.com/paper/2404.08080/) (2024)
- [Learning Quantile Functions without Quantile Crossing for Distribution-free Time Series Forecasting](https://mlsanity.com/paper/2111.06581/) (2021)
- [Multivariate Quantile Function Forecaster](https://mlsanity.com/paper/2202.11316/) (2022)
- [GraB: Finding Provably Better Data Permutations than Random Reshuffling](https://mlsanity.com/paper/2205.10733/) (2022)
- [Robust Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2202.11910/) (2022)
- [Coordinating Distributed Example Orders for Provably Accelerated Training](https://mlsanity.com/paper/2302.00845/) (2023)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [Variance Matters: Improving Domain Adaptation via Stratified Sampling](https://mlsanity.com/paper/2512.05226/) (2025)
- [Adaptive Sampling for Probabilistic Forecasting under Distribution Shift](https://mlsanity.com/paper/2302.11870/) (2023)
- [Order Matters: Improving Domain Adaptation by Reordering Data](https://mlsanity.com/paper/2605.05084/) (2026)
- [Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMs](https://mlsanity.com/paper/2512.21915/) (2025)
- [Variance-reduced Domain Adaptation using Paired Sampling](https://mlsanity.com/paper/2607.20367/) (2026)
- [Online Variance Reduction for Domain Adaptation on Streaming Data](https://mlsanity.com/paper/2607.20374/) (2026)
- [Kinematics-Induced Multimodal 3D Human Pose Estimation with Subject-Level Privacy](https://mlsanity.com/paper/2610.02943/) (2026)

## Abstract (arXiv)

In large-scale time series forecasting, one often encounters the situation where the temporal patterns of time series, while drifting over time, differ from one another in the same dataset. In this paper, we provably show under such heterogeneity, training a forecasting model with commonly used stochastic optimizers (e.g. SGD) potentially suffers large variance on gradient estimation, and thus incurs long-time training. We show that this issue can be efficiently alleviated via stratification, which allows the optimizer to sample from pre-grouped time series strata. For better trading-off gradient variance and computation complexity, we further propose SCott (Stochastic Stratified Control Variate Gradient Descent), a variance reduced SGD-style optimizer that utilizes stratified sampling via control variate. In theory, we provide the convergence guarantee of SCott on smooth non-convex objectives. Empirically, we evaluate SCott and other baseline optimizers on both synthetic and real-world time series forecasting problems, and demonstrate SCott converges faster with respect to both iterations and wall clock time.