# Hybrid Acceleration Scheme for Variance Reduced Stochastic Optimization Algorithms

arXiv: https://arxiv.org/abs/2111.06791 · HTML: https://mlsanity.com/paper/2111.06791/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2021.

## Built on by (most cited first, 1 of 1)

- [SPIRAL: A superlinearly convergent incremental proximal algorithm for nonconvex finite sum minimization](https://mlsanity.com/paper/2207.08195/) (2022)

## Abstract (arXiv)

Stochastic variance reduced optimization methods are known to be globally convergent while they suffer from slow local convergence, especially when moderate or high accuracy is needed. To alleviate this problem, we propose an optimization algorithm -- which we refer to as a hybrid acceleration scheme -- for a class of proximal variance reduced stochastic optimization algorithms. The proposed optimization scheme combines a fast locally convergent algorithm, such as a quasi--Newton method, with a globally convergent variance reduced stochastic algorithm, for instance SAGA or L--SVRG. Our global convergence result of the hybrid acceleration method is based on specific safeguard conditions that need to be satisfied for a step of the locally fast convergent method to be accepted. We prove that the sequence of the iterates generated by the hybrid acceleration scheme converges almost surely to a solution of the underlying optimization problem. We also provide numerical experiments that show significantly improved convergence of the hybrid acceleration scheme compared to the basic stochastic variance reduced optimization algorithm.