mlsanityData to 7 Oct 2026

Source-Optimal Training is Transfer-Suboptimal

arXiv 2511.08401 ↗First posted 11 Nov 2025stat.ML cs.LG math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

We prove that training a source model optimally for its own task is generically suboptimal when the objective is downstream transfer. We study the source-side optimization problem in L2-SP ridge regression and show a fundamental mismatch between the source-optimal and transfer-optimal source regularization: outside of a measure-zero set, τ0^* neq τS^*. We characterize the transfer-optimal source penalty τ0^* as a function of task alignment and identify an alignment-dependent reversal: with imperfect alignment (0<ρ<1), transfer benefits from stronger source regularization, while in super-aligned regimes (ρ>1), transfer benefits from weaker regularization. Additionally, in isotropic settings, the decision of whether transfer helps is independent of the target sample size and noise, depending only on task alignment and source characteristics. We verify the linear predictions in a synthetic ridge regression experiment, and we present experiments on MNIST, CIFAR-10, and 20 Newsgroups as evidence that the source-optimal versus transfer-optimal mismatch persists in standard nonlinear transfer learning pipelines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

14 of 19 references.

References · 14

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  5. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  6. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  7. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  8. 2025Overcoming catastrophic forgetting in neural networkscited by 52
  9. 2020Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transferscited by 26
  10. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 21
  11. 2020Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Taskscited by 15
  12. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 10
2 more
  1. 2021The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regressioncited by 6
  2. 2024Universality in Transfer Learning for Linear Modelscited by 6

Built on by

Not cited yet.