mlsanityData to 14 Oct 2026

Factor Augmented High-Dimensional SGD

arXiv 2605.19291 ↗First posted 19 May 2026stat.ML cs.LG math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations in high-dimensional learning tasks. Unlike standard two-stage dimension reduction approaches that rely on offline representation learning and full data storage, a key novelty of FSGD is that it operates purely on streaming data, making it scalable to large-scale and high-dimensional problems. Furthermore, we establish the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, and provide moment convergence in ℓs norm under decaying step sizes and mini-batch updates. Our results provide a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

34 of 56 references.

References · 34

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  6. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  7. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  8. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  9. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  10. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
  11. 2016First Efficient Convergence for Streaming k-PCA: a Global, Gap-Free, and Near-Optimal Ratecited by 32
  12. 2022Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regressioncited by 31
22 more
  1. 2016Streaming PCA: Matching Matrix Bernstein and Near-Optimal Finite Sample Guarantees for Oja's Algorithmcited by 29
  2. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 27
  3. 2023Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noisecited by 26
  4. 2022DP-PCA: Statistically Optimal and Differentially Private PCAcited by 24
  5. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  6. 2021Streaming k-PCA: Efficient guarantees for Oja's algorithm, beyond rank-one updatescited by 21
  7. 2015Sufficient Forecasting Using Factor Modelscited by 13
  8. 2019Stochastic Subspace Descentcited by 10
  9. 2021CP Factor Model for Dynamic Tensorscited by 10
  10. 2017Inverse Moment Methods for Sufficient Forecasting using High-Dimensional Predictorscited by 9
  11. 2020Tensor Factor Model Estimation by Iterative Projectioncited by 9
  12. 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 9
  13. 2024Factor Augmented Tensor-on-Tensor Neural Networkscited by 7
  14. 2021Simultaneous Decorrelation of Matrix Time Seriescited by 5
  15. 2024Estimation and Inference for CP Tensor Factor Modelscited by 4
  16. 2024Dynamic Matrix Factor Models for High Dimensional Time Seriescited by 4
  17. 2025Covariate-Adjusted Deep Causal Learning for Heterogeneous Panel Data Modelscited by 2
  18. 2025Statistical Guarantees for High-Dimensional Stochastic Gradient Descentcited by 2
  19. 2025Modewise Additive Factor Model for Matrix Time Seriescited by 2
  20. 2025Supervised Dynamic Dimension Reduction with Deep Neural Networkcited by 1
  21. 2025Factor Augmented Supervised Learning with Text Embeddingscited by 1
  22. 2025Diffusion Index Forecasting with Tensor Datacited by 1

Built on by

Not cited yet.