# Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors

arXiv: https://arxiv.org/abs/2002.10614 · HTML: https://mlsanity.com/paper/2002.10614/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2020.

## Built on by (most cited first, 11 of 11)

- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression](https://mlsanity.com/paper/2103.05621/) (2021)
- [Double Descent and Other Interpolation Phenomena in GANs](https://mlsanity.com/paper/2106.04003/) (2021)
- [TL-PCA: Transfer Learning of Principal Component Analysis](https://mlsanity.com/paper/2410.10805/) (2024)
- [Does Double Descent Occur in Self-Supervised Learning?](https://mlsanity.com/paper/2307.07872/) (2023)
- [How Much Training Data is Memorized in Overparameterized Autoencoders? An Inverse Problem Perspective on Memorization Evaluation](https://mlsanity.com/paper/2310.02897/) (2023)
- [Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networks](https://mlsanity.com/paper/2103.00500/) (2021)
- [Understanding the Role of Optimization in Double Descent](https://mlsanity.com/paper/2312.03951/) (2023)
- [Investigating the Impact of Model Complexity in Large Language Models](https://mlsanity.com/paper/2410.00699/) (2024)
- [On the Relationship Between Double Descent of CNNs and Shape/Texture Bias Under Learning Process](https://mlsanity.com/paper/2503.02302/) (2025)
- [Stochastic Trust-Region Methods for Over-parameterized Models](https://mlsanity.com/paper/2604.14017/) (2026)

## Abstract (arXiv)

We study the linear subspace fitting problem in the overparameterized setting, where the estimated subspace can perfectly interpolate the training examples. Our scope includes the least-squares solutions to subspace fitting tasks with varying levels of supervision in the training data (i.e., the proportion of input-output examples of the desired low-dimensional mapping) and orthonormality of the vectors defining the learned operator. This flexible family of problems connects standard, unsupervised subspace fitting that enforces strict orthonormality with a corresponding regression task that is fully supervised and does not constrain the linear operator structure. This class of problems is defined over a supervision-orthonormality plane, where each coordinate induces a problem instance with a unique pair of supervision level and softness of orthonormality constraints. We explore this plane and show that the generalization errors of the corresponding subspace fitting problems follow double descent trends as the settings become more supervised and less orthonormally constrained.