# Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction

arXiv: https://arxiv.org/abs/2403.12511 · HTML: https://mlsanity.com/paper/2403.12511/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 14 of 14)

- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [An Improved Analysis of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2007.07989/) (2020)
- [Memory-Efficient Backpropagation Through Time](https://mlsanity.com/paper/1606.03401/) (2016)
- [Accelerating Federated Learning via Momentum Gradient Descent](https://mlsanity.com/paper/1910.03197/) (2019)
- [Gradients without Backpropagation](https://mlsanity.com/paper/2202.08587/) (2022)
- [Convergence Rate of Frank-Wolfe for Non-Convex Objectives](https://mlsanity.com/paper/1607.00345/) (2016)
- [On the insufficiency of existing momentum schemes for Stochastic Optimization](https://mlsanity.com/paper/1803.05591/) (2018)
- [Reversible Vision Transformers](https://mlsanity.com/paper/2302.04869/) (2023)
- [Can Forward Gradient Match Backpropagation?](https://mlsanity.com/paper/2306.06968/) (2023)
- [Deep Neural Network Training with Frank-Wolfe](https://mlsanity.com/paper/2010.07243/) (2020)
- [Compression-aware Training of Neural Networks using Frank-Wolfe](https://mlsanity.com/paper/2205.11921/) (2022)
- [Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction](https://mlsanity.com/paper/2202.00441/) (2022)
- [Gradient descent with momentum --- to accelerate or to super-accelerate?](https://mlsanity.com/paper/2001.06472/) (2020)
- [Moonwalk: Inverse-Forward Differentiation](https://mlsanity.com/paper/2402.14212/) (2024)

## Built on by (most cited first, 5 of 5)

- [FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative](https://mlsanity.com/paper/2512.20814/) (2025)
- [Memory Savings at What Cost? A Study of Alternatives to Backpropagation](https://mlsanity.com/paper/2506.21833/) (2025)
- [FedScalar: Federated Learning with Scalar Communication for Bandwidth-Constrained Networks](https://mlsanity.com/paper/2410.02260/) (2024)
- [Time-Varying Convex Optimization with O(n) Computational Complexity](https://mlsanity.com/paper/2410.15009/) (2024)
- [Scalar Federated Learning for Linear Quadratic Regulator](https://mlsanity.com/paper/2604.05088/) (2026)

## Abstract (arXiv)

This paper aims to enhance the use of the Frank-Wolfe (FW) algorithm for training deep neural networks. Similar to any gradient-based optimization algorithm, FW suffers from high computational and memory costs when computing gradients for DNNs. This paper introduces the application of the recently proposed projected forward gradient (Projected-FG) method to the FW framework, offering reduced computational cost similar to backpropagation and low memory utilization akin to forward propagation. Our results show that trivial application of the Projected-FG introduces non-vanishing convergence error due to the stochastic noise that the Projected-FG method introduces in the process. This noise results in an non-vanishing variance in the Projected-FG estimated gradient. To address this, we propose a variance reduction approach by aggregating historical Projected-FG directions. We demonstrate rigorously that this approach ensures convergence to the optimal solution for convex functions and to a stationary point for non-convex functions. These convergence properties are validated through a numerical example, showcasing the approach's effectiveness and efficiency.