mlsanityData to 7 Oct 2026

Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them

arXiv 2510.19634 ↗First posted 22 Oct 2025cs.LG cs.NA math.NAcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper argues that the method of least squares has significant unfulfilled potential in modern machine learning, far beyond merely being a tool for fitting linear models. To release its potential, we derive custom gradients that transform the solver into a differentiable operator, like a neural network layer, enabling many diverse applications. Empirically, we demonstrate: (i) scalability by enforcing weight sparsity on a 50 million parameter model; (ii) imposing conservativeness constraints in score-based generative models; and (iii) hyperparameter tuning of Gaussian processes based on predictive performance. By doing this, our work represents the next iteration in developing differentiable linear-algebra tools and making them widely accessible to machine learning practitioners.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 38 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

19 of 38 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2016Group Equivariant Convolutional Networkscited by 1,154
  6. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  7. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  9. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  10. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
  11. 2021DC3: A learning method for optimization with hard constraintscited by 181
  12. 2021Frame Averaging for Invariant and Equivariant Network Designcited by 166
7 more
  1. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 143
  2. 2024The Elements of Differentiable Programmingcited by 41
  3. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  4. 2022Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraintscited by 25
  5. 2022On Investigating the Conservative Property of Score-Based Generative Modelscited by 16
  6. 2023JaxPruner: A concise library for sparsity researchcited by 13
  7. 2024Gradients of Functions of Large Matricescited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025VIKING: Deep variational inference with stochastic projectionscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
  2. 2024On the Crucial Role of Initialization for Matrix Factorization
  3. 2023Mixed Regression via Approximate Message Passing
  4. 2026Free Decompression with Algebraic Spectral Curves
  5. 2026lapanda: A Matrix-Free Differentiable Solver for Nonconvex Constrained Optimization Layers
  6. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization
  7. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
  8. 2025Optimization-Free Diffusion Model -- A Perturbation Theory Approach