# Learning with SGD and Random Features

arXiv: https://arxiv.org/abs/1807.06343 · HTML: https://mlsanity.com/paper/1807.06343/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2018.

## Built on by (most cited first, 30 of 33)

- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality](https://mlsanity.com/paper/2106.08900/) (2021)
- [Kernel Continual Learning](https://mlsanity.com/paper/2107.05757/) (2021)
- [Universal approximation property of Banach space-valued random feature models including random neural networks](https://mlsanity.com/paper/2312.08410/) (2023)
- [Non-Parametric Representation Learning with Kernels](https://mlsanity.com/paper/2309.02028/) (2023)
- [How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent](https://mlsanity.com/paper/2309.08044/) (2023)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [Random Fourier Signature Features](https://mlsanity.com/paper/2311.12214/) (2023)
- [A spectral-based analysis of the separation between two-layer neural networks and linear methods](https://mlsanity.com/paper/2108.04964/) (2021)
- [Random features models: a way to study the success of naive imputation](https://mlsanity.com/paper/2402.03839/) (2024)
- [Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models](https://mlsanity.com/paper/2502.05074/) (2025)
- [ParK: Sound and Efficient Kernel Ridge Regression by Feature Space Partitions](https://mlsanity.com/paper/2106.12231/) (2021)
- [On the Double Descent of Random Features Models Trained with SGD](https://mlsanity.com/paper/2110.06910/) (2021)
- [Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization](https://mlsanity.com/paper/2310.01139/) (2023)
- [Exponential Error Convergence in Data Classification with Optimized Random Features: Acceleration by Quantum Machine Learning](https://mlsanity.com/paper/2106.09028/) (2021)
- [Universal randomised signatures for generative time series modelling](https://mlsanity.com/paper/2406.10214/) (2024)
- [Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/2407.07670/) (2024)
- [Optimal Kernel Quantile Learning with Random Features](https://mlsanity.com/paper/2408.13591/) (2024)
- [Random Feature Representation Boosting](https://mlsanity.com/paper/2501.18283/) (2025)
- [RandNet-Parareal: a time-parallel PDE solver using Random Neural Networks](https://mlsanity.com/paper/2411.06225/) (2024)
- [Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data](https://mlsanity.com/paper/2511.18661/) (2025)
- [MetaKernel: Learning Variational Random Features with Limited Labels](https://mlsanity.com/paper/2105.03781/) (2021)
- [Coefficient-based Regularized Distribution Regression](https://mlsanity.com/paper/2208.12427/) (2022)
- [Random feature approximation for general spectral methods](https://mlsanity.com/paper/2506.16283/) (2025)
- [SchoenbAt: Rethinking Attention with Polynomial basis](https://mlsanity.com/paper/2505.12252/) (2025)
- [Shallow Representation is Deep: Learning Uncertainty-aware and Worst-case Random Feature Dynamics](https://mlsanity.com/paper/2106.13066/) (2021)
- [Ridgeless Regression with Random Features](https://mlsanity.com/paper/2205.00477/) (2022)
- [Random feature approximation for general spectral methods](https://mlsanity.com/paper/2308.15434/) (2023)
- [Macformer: Transformer with Random Maclaurin Feature Attention](https://mlsanity.com/paper/2408.11656/) (2024)
- [Random Neural Network Expressivity for Non-Linear Partial Differential Equations](https://mlsanity.com/paper/2605.25057/) (2026)

## Abstract (arXiv)

Sketching and stochastic gradient methods are arguably the most common techniques to derive efficient large scale learning algorithms. In this paper, we investigate their application in the context of nonparametric statistical learning. More precisely, we study the estimator defined by stochastic gradient with mini batches and random features. The latter can be seen as form of nonlinear sketching and used to define approximate kernel methods. The considered estimator is not explicitly penalized/constrained and regularization is implicit. Indeed, our study highlights how different parameters, such as number of features, iterations, step-size and mini-batch size control the learning properties of the solutions. We do this by deriving optimal finite sample bounds, under standard assumptions. The obtained results are corroborated and illustrated by numerical experiments.