# Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection

arXiv: https://arxiv.org/abs/2007.13171 · HTML: https://mlsanity.com/paper/2007.13171/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2020.

## Built on by (most cited first, 12 of 12)

- [Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies](https://mlsanity.com/paper/2306.17648/) (2023)
- [Learning Regularization Parameters of Inverse Problems via Deep Neural Networks](https://mlsanity.com/paper/2104.06594/) (2021)
- [A Method for Computing Inverse Parametric PDE Problems with Random-Weight Neural Networks](https://mlsanity.com/paper/2210.04338/) (2022)
- [Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks](https://mlsanity.com/paper/2201.09989/) (2022)
- [slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks](https://mlsanity.com/paper/2109.14002/) (2021)
- [Optimal Matrix-Mimetic Tensor Algebras via Variable Projection](https://mlsanity.com/paper/2406.06942/) (2024)
- [Online Learning Under A Separable Stochastic Approximation Framework](https://mlsanity.com/paper/2305.07484/) (2023)
- [Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling](https://mlsanity.com/paper/2507.22045/) (2025)
- [A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization](https://mlsanity.com/paper/2511.01234/) (2025)
- [Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective](https://mlsanity.com/paper/2601.07397/) (2026)
- [Constrained Variable Projection for Structured Problems](https://mlsanity.com/paper/2606.23939/) (2026)
- [Initialization-Free Bundle Adjustment Revisited: A Controlled Experimental Study](https://mlsanity.com/paper/2608.18028/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have achieved state-of-the-art performance across a variety of traditional machine learning tasks, e.g., speech recognition, image classification, and segmentation. The ability of DNNs to efficiently approximate high-dimensional functions has also motivated their use in scientific applications, e.g., to solve partial differential equations (PDE) and to generate surrogate models. In this paper, we consider the supervised training of DNNs, which arises in many of the above applications. We focus on the central problem of optimizing the weights of the given DNN such that it accurately approximates the relation between observed input and target data. Devising effective solvers for this optimization problem is notoriously challenging due to the large number of weights, non-convexity, data-sparsity, and non-trivial choice of hyperparameters. To solve the optimization problem more efficiently, we propose the use of variable projection (VarPro), a method originally designed for separable nonlinear least-squares problems. Our main contribution is the Gauss-Newton VarPro method (GNvpro) that extends the reach of the VarPro idea to non-quadratic objective functions, most notably, cross-entropy loss functions arising in classification. These extensions make GNvpro applicable to all training problems that involve a DNN whose last layer is an affine mapping, which is common in many state-of-the-art architectures. In our four numerical experiments from surrogate modeling, segmentation, and classification GNvpro solves the optimization problem more efficiently than commonly-used stochastic gradient descent (SGD) schemes. Also, GNvpro finds solutions that generalize well, and in all but one example better than well-tuned SGD methods, to unseen data points.