mlsanityData to 7 Oct 2026

A Progressive Batching L-BFGS Method for Machine Learning

arXiv 1802.05374 ↗First posted 15 Feb 2018math.OC cs.LG stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objective function. All of this appears to call for a full batch approach, but since small batch sizes give rise to faster algorithms with better generalization properties, L-BFGS is currently not considered an algorithm of choice for large-scale machine learning applications. One need not, however, choose between the two extremes represented by the full batch or highly stochastic regimes, and may instead follow a progressive batching approach in which the sample size increases during the course of the optimization. In this paper, we present a new version of the L-BFGS algorithm that combines three basic components - progressive batching, a stochastic line search, and stable quasi-Newton updating - and that performs well on training logistic regression and deep neural networks. We provide supporting convergence theory for the method.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 7; 2023: 8; 2024: 8; 2025: 5; 2026: 6.

45 papers.

Sort:

By area:

  1. 2022A Survey of Machine Unlearningcited by 298
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 109
  4. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  5. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  6. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  7. 2021EventHPE: Event-based 3D Human Pose and Shape Estimationcited by 42
  8. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  9. 2023Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Modelscited by 15
  10. 2021Better SGD using Second-order Momentumcited by 14
  11. 2021Human Pose and Shape Estimation from Single Polarization Imagescited by 13
  12. 2026Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networkscited by 11
33 more
  1. 2023Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Groupingcited by 10
  2. 2023Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtrackingcited by 9
  3. 2022SketchySGD: Reliable Stochastic Optimization via Randomized Curvature Estimatescited by 8
  4. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  5. 2021Stochastic Gradient Line Bayesian Optimization for Efficient Noise-Robust Optimization of Parameterized Quantum Circuitscited by 7
  6. 2021Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspacescited by 5
  7. 2021Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimizationcited by 4
  8. 2023A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systemscited by 4
  9. 2025SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learningcited by 4
  10. 2024Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradientscited by 3
  11. 2021Quasi-Newton Quasi-Monte Carlo for variational Bayescited by 2
  12. 2021A Retrospective Approximation Approach for Smooth Stochastic Optimizationcited by 1
  13. 2021Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learningcited by 1
  14. 2022On the efficiency of Stochastic Quasi-Newton Methods for Deep Learningcited by 1
  15. 2024Faster Convergence for Transformer Fine-tuning with Line Search Methodscited by 1
  16. 2024Improving Line Search Methods for Large Scale Neural Network Trainingcited by 1
  17. 2024No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptationcited by 1
  18. 2024Derivative-Free Optimization via Finite Difference Approximation: An Experimental Studycited by 1
  19. 2025Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stabilitycited by 1
  20. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
  21. 2021Exact Stochastic Second Order Deep Learningcited by 0
  22. 2022Hyper-Learning for Gradient-Based Batch Size Adaptationcited by 0
  23. 2022Causal Inference via Nonlinear Variable Decorrelation for Healthcare Applicationscited by 0
  24. 2023Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Ratecited by 0
  25. 2023Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimizationcited by 0
  26. 2023Information-Theoretic Trust Regions for Stochastic Gradient-Based Optimizationcited by 0
  27. 2024A model learning framework for inferring the dynamics of transmission rate depending on exogenous variables for epidemic forecastscited by 0
  28. 2025Enhanced Derivative-Free Optimization Using Adaptive Correlation-Induced Finite Difference Estimatorscited by 0
  29. 2025Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usagecited by 0
  30. 2026Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descentcited by 0
  31. 2026Accelerated Gradient Descent for Faster Convergence with Minimal Overheadcited by 0
  32. 2026FACT: A Simple and Efficient Framework for Active Finetuningcited by 0
  33. 2026Benchmarking Optimizers to Solve Inverse Problems with Differentiable Physics Simulatorscited by 0