mlsanityData to 7 Oct 2026

A Multi-Batch L-BFGS Method for Machine Learning

arXiv 1605.06049 ↗First posted 19 May 2016math.OC cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each iteration to facilitate parallelism, and that employ second-order information. In order to improve the learning process, we follow a multi-batch approach in which the batch changes at each iteration. This can cause difficulties because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, illustrates the behavior of the algorithm in a distributed computing platform, and studies its convergence properties for both the convex and nonconvex cases.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 4; 2024: 3; 2025: 6; 2026: 4.

30 papers.

Sort:

By area:

  1. 2022A Survey of Machine Unlearningcited by 298
  2. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 109
  3. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  4. 2025Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networkscited by 41
  5. 2022On amortizing convex conjugates for optimal transportcited by 30
  6. 2022Fast Bayesian Inference with Batch Bayesian Quadrature via Kernel Recombinationcited by 20
  7. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  8. 2022SP2: A Second Order Stochastic Polyak Methodcited by 13
  9. 2021Resource-constrained Federated Edge Learning with Heterogeneous Data: Formulation and Analysiscited by 10
  10. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  11. 2022Deep Leakage from Model in Federated Learningcited by 9
  12. 2024Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attentioncited by 5
18 more
  1. 2021Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimizationcited by 4
  2. 2023A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systemscited by 4
  3. 2025SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learningcited by 4
  4. 2024Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradientscited by 3
  5. 2021Quasi-Newton Quasi-Monte Carlo for variational Bayescited by 2
  6. 2023Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methodscited by 2
  7. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  8. 2021A Retrospective Approximation Approach for Smooth Stochastic Optimizationcited by 1
  9. 2022On the efficiency of Stochastic Quasi-Newton Methods for Deep Learningcited by 1
  10. 2025Mobile Robotic Multi-View Photometric Stereocited by 1
  11. 2025Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stabilitycited by 1
  12. 2022Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimizationcited by 0
  13. 2023Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimizationcited by 0
  14. 2025Knowledge Distillation with Adapted Weightcited by 0
  15. 2025FUSE: First-Order and Second-Order Unified SynthEsis in Stochastic Optimizationcited by 0
  16. 2026Second-Order, First-Class: A Composable Stack for Curvature-Aware Trainingcited by 0
  17. 2026A Weighted Kernel Method for Approximation that Adapts to Learned Multivariable Structurecited by 0
  18. 2026SoftServe: A Scalable Quasi-Newton Method for Deep Learningcited by 0