# An Element-wise RSAV Algorithm for Unconstrained Optimization Problems

arXiv: https://arxiv.org/abs/2309.04013 · HTML: https://mlsanity.com/paper/2309.04013/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 37 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [AEGD: Adaptive Gradient Descent with Energy](https://mlsanity.com/paper/2010.05109/) (2020)
- [Stochastic Steffensen method](https://mlsanity.com/paper/2211.15310/) (2022)
- [An efficient and robust SAV based algorithm for discrete gradient systems arising from optimizations](https://mlsanity.com/paper/2301.02942/) (2023)

## Abstract (arXiv)

We present a novel optimization algorithm, element-wise relaxed scalar auxiliary variable (E-RSAV), that satisfies an unconditional energy dissipation law and exhibits improved alignment between the modified and the original energy. Our algorithm features rigorous proofs of linear convergence in the convex setting. Furthermore, we present a simple accelerated algorithm that improves the linear convergence rate to super-linear in the univariate case. We also propose an adaptive version of E-RSAV with Steffensen step size. We validate the robustness and fast convergence of our algorithm through ample numerical experiments.