mlsanityData to 14 Oct 2026

Adaptive Sampling Strategies for Stochastic Optimization

arXiv 1710.11258 ↗First posted 30 Oct 2017math.OC stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by increasing the sample size as needed. The decision to increase the sample size is governed by an inner product test that ensures that search directions are descent directions with high probability. We show that the inner product test improves upon the well known norm test, and can be used as a basis for an algorithm that is globally convergent on nonconvex functions and enjoys a global linear rate of convergence on strongly convex functions. Numerical experiments on logistic regression problems illustrate the performance of the algorithm.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 3; 2023: 3; 2024: 5; 2025: 3; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  2. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  3. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  4. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
  5. 2021An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangianscited by 25
  6. 2021Inequality Constrained Stochastic Nonlinear Optimization via Active-Set Sequential Quadratic Programmingcited by 17
  7. 2021Globally Convergent Multilevel Training of Deep Residual Networkscited by 11
  8. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 9
  9. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  10. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  11. 2021Stochastic Gradient Line Bayesian Optimization for Efficient Noise-Robust Optimization of Parameterized Quantum Circuitscited by 7
  12. 2025IMPACT: A Generic Semantic Loss for Multimodal Medical Image Registrationcited by 5
18 more
  1. 2021Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimizationcited by 4
  2. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  3. 2024Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelismcited by 4
  4. 2024AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Sizecited by 3
  5. 2023Communication-Efficient Federated Optimization over Semi-Decentralized Networkscited by 2
  6. 2026Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descentcited by 2
  7. 2021A Retrospective Approximation Approach for Smooth Stochastic Optimizationcited by 1
  8. 2021Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learningcited by 1
  9. 2025Efficient Personalization of Quantized Diffusion Model without Backpropagationcited by 1
  10. 2021Discriminative Bayesian filtering lends momentum to the stochastic Newton method for minimizing log-convex functionscited by 0
  11. 2021GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networkscited by 0
  12. 2021On the equivalence of different adaptive batch size selection strategies for stochastic gradient descent methodscited by 0
  13. 2021Bolstering Stochastic Gradient Descent with Model Buildingcited by 0
  14. 2022A Stochastic Proximal Method for Nonsmooth Regularized Finite Sum Optimizationcited by 0
  15. 2022Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothnesscited by 0
  16. 2025AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Modelscited by 0
  17. 2026Guiding the Recommender: Information-Aware Auto-Bidding for Content Promotioncited by 0
  18. 2026Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variancecited by 0