mlsanityData to 7 Oct 2026

Constrained Variable Projection for Structured Problems

arXiv 2606.23939 ↗First posted 22 Jun 2026math.OC cs.LG cs.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

Variable projection is a classical technique for separable nonlinear least-squares problems, in which variables that enter linearly are eliminated exactly, yielding a reduced nonlinear problem. By expressing this framework as a particular instance of a broader class of bilevel optimization problems, we develop a constrained variable-projection framework for data-science models, where the remaining variables are subject to convex constraints and the eliminated variables arise from a lower-level least-squares problem. In particular, by interpreting variable projection as a collapsed bilevel optimization problem, we derive exact reduced-gradient formulas compatible with automatic differentiation and propose a conditional-gradient algorithm for the resulting constrained reduced problem. We establish convergence guarantees under standard smoothness and compactness assumptions, and discuss extensions to structured lower-level variables. Numerical experiments on sparse autoencoding, dictionary learning, blind deconvolution, and few-shot learning suggest that the method can improve wall-clock efficiency and data efficiency relative to natural joint-optimization baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

23 of 55 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  6. 2019Differentiable Convex Optimization Layerscited by 458
  7. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  8. 2021Efficient and Modular Implicit Differentiationcited by 199
  9. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  10. 2018Recent Advances in Autoencoder-Based Representation Learningcited by 130
  11. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  12. 2016On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimizationcited by 108
11 more
  1. 2021Complexity of Linear Minimization and Projection on Some Setscited by 19
  2. 2020Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projectioncited by 12
  3. 2022Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networkscited by 8
  4. 2021slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networkscited by 6
  5. 2013An Equivalence between the Lasso and Support Vector Machinescited by 4
  6. 2019Active set complexity of the Away-step Frank-Wolfe Algorithmcited by 4
  7. 2021Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimatescited by 4
  8. 2022Modeling Dynamic User Preference via Dictionary Learning for Sequential Recommendationcited by 4
  9. 2024Variable Projection Algorithms: Theoretical Insights and A Novel Approach for Problems with Large Residualcited by 3
  10. 2013Variable projection methods for approximate (greatest) common divisor computationscited by 2
  11. 2026Grassmannian Geometry and Global Convergence of Variable Projection for Neural Networkscited by 1

Built on by

Not cited yet.