mlsanityData to 14 Oct 2026

FUSE: First-Order and Second-Order Unified SynthEsis in Stochastic Optimization

arXiv 2503.04204 ↗First posted 6 Mar 2025cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic optimization methods have actively been playing a critical role in modern machine learning algorithms to deliver decent performance. While numerous works have proposed and developed diverse approaches, first-order and second-order methods are in entirely different situations. The former is significantly pivotal and dominating in emerging deep learning but only leads convergence to a stationary point. However, second-order methods are less popular due to their computational intensity in large-dimensional problems. This paper presents a novel method that leverages both the first-order and second-order methods in a unified algorithmic framework, termed FUSE, from which a practical version (PV) is derived accordingly. FUSE-PV stands as a simple yet efficient optimization method involving a switch-over between first and second orders. Additionally, we develop different criteria that determine when to switch. FUSE-PV has provably shown a smaller computational complexity than SGD and Adam. To validate our proposed scheme, we present an ablation study on several simple test functions and show a comparison with baselines for benchmark datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

19 of 27 references.

References · 19

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  2. 2021Convex optimizationcited by 2,515
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  5. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  6. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  7. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  8. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  9. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 40
  10. 2017Second-Order Optimization for Non-Convex Machine Learning: An Empirical Studycited by 35
  11. 2016A Multi-Batch L-BFGS Method for Machine Learningcited by 31
  12. 2020Scalable Second Order Optimization for Deep Learningcited by 21
7 more
  1. 2020Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimizationcited by 16
  2. 2020AdaSGD: Bridging the gap between SGD and Adamcited by 8
  3. 2020Mixing ADAM and SGD: a Combined Optimization Methodcited by 8
  4. 2020LSOS: Line-search Second-Order Stochastic optimization methods for nonconvex finite sumscited by 4
  5. 2023mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimizationcited by 4
  6. 2019On the Convergence of AdaBound and its Connection to SGDcited by 3
  7. 2021Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive Sample Size Approachcited by 2

Built on by

Not cited yet.