# Deep Neural Network Training with Frank-Wolfe

arXiv: https://arxiv.org/abs/2010.07243 · HTML: https://mlsanity.com/paper/2010.07243/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [Training Deep Learning Models with Norm-Constrained LMOs](https://mlsanity.com/paper/2502.07529/) (2025)
- [Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise](https://mlsanity.com/paper/2506.04192/) (2025)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search](https://mlsanity.com/paper/2102.11646/) (2021)
- [Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging](https://mlsanity.com/paper/2306.16788/) (2023)
- [Compression-aware Training of Neural Networks using Frank-Wolfe](https://mlsanity.com/paper/2205.11921/) (2022)
- [PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs](https://mlsanity.com/paper/2312.15230/) (2023)
- [Interpretability Guarantees with Merlin-Arthur Classifiers](https://mlsanity.com/paper/2206.00759/) (2022)
- [Approximate Vanishing Ideal Computations at Scale](https://mlsanity.com/paper/2207.01236/) (2022)
- [Interpretable Neural Networks with Frank-Wolfe: Sparse Relevance Maps and Relevance Orderings](https://mlsanity.com/paper/2110.08105/) (2021)
- [Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction](https://mlsanity.com/paper/2403.12511/) (2024)
- [Approximating Latent Manifolds in Neural Networks via Vanishing Ideals](https://mlsanity.com/paper/2502.15051/) (2025)
- [How much pre-training is enough to discover a good subnetwork?](https://mlsanity.com/paper/2108.00259/) (2021)
- [Training Characteristic Functions with Reinforcement Learning: XAI-methods play Connect Four](https://mlsanity.com/paper/2202.11797/) (2022)
- [EsaCL: Efficient Continual Learning of Sparse Models](https://mlsanity.com/paper/2401.05667/) (2024)
- [Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2404.09438/) (2024)
- [Revisiting Frank-Wolfe for Structured Nonconvex Optimization](https://mlsanity.com/paper/2503.08921/) (2025)
- [Don't Be Greedy, Just Relax! Pruning LLMs via Frank-Wolfe](https://mlsanity.com/paper/2510.13713/) (2025)
- [i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery](https://mlsanity.com/paper/2112.04905/) (2021)
- [Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture](https://mlsanity.com/paper/2501.16322/) (2025)
- [Projection-Free CNN Pruning via Frank-Wolfe with Momentum: Sparser Models with Less Pretraining](https://mlsanity.com/paper/2512.01147/) (2025)
- [Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent](https://mlsanity.com/paper/2605.08850/) (2026)
- [Local LMO is Secretly a Projection Method!](https://mlsanity.com/paper/2609.33383/) (2026)

## Abstract (arXiv)

This paper studies the empirical efficacy and benefits of using projection-free first-order methods in the form of Conditional Gradients, a.k.a. Frank-Wolfe methods, for training Neural Networks with constrained parameters. We draw comparisons both to current state-of-the-art stochastic Gradient Descent methods as well as across different variants of stochastic Conditional Gradients. In particular, we show the general feasibility of training Neural Networks whose parameters are constrained by a convex feasible region using Frank-Wolfe algorithms and compare different stochastic variants. We then show that, by choosing an appropriate region, one can achieve performance exceeding that of unconstrained stochastic Gradient Descent and matching state-of-the-art results relying on L2-regularization. Lastly, we also demonstrate that, besides impacting performance, the particular choice of constraints can have a drastic impact on the learned representations.