mlsanityData to 14 Oct 2026

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift

arXiv 2208.01857 ↗First posted 3 Aug 2022cs.LG math.OC stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

We study linear regression under covariate shift, where the marginal distribution over the input covariates differs in the source and the target domains, while the conditional distribution of the output given the input covariates is similar across the two domains. We investigate a transfer learning approach with pretraining on the source data and finetuning based on the target data (both conducted by online SGD) for this problem. We establish sharp instance-dependent excess risk upper and lower bounds for this approach. Our bounds suggest that for a large class of linear regression instances, transfer learning with O(N2) source data (and scarce or no target data) is as effective as supervised learning with N target data. In addition, we show that finetuning, even with only a small amount of target data, could drastically reduce the amount of source data required by pretraining. Our theory sheds light on the effectiveness and limitation of pretraining as well as the benefits of finetuning for tackling covariate shift problems.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 3 Aug 2022.

Builds on

19 of 40 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Invariant Risk Minimizationcited by 1,854
  3. 2014How transferable are features in deep neural networks?cited by 1,225
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2020Benign overfitting in ridge regressioncited by 153
  6. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  7. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  8. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  9. 2018Marginal Singularity, and the Benefits of Labels in Covariate-Shiftcited by 65
  10. 2020On the Value of Target Data in Transfer Learningcited by 65
  11. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  12. 2019Domain Adaptation with Asymmetrically-Relaxed Distribution Alignmentcited by 61
7 more
  1. 2022Optimally tackling covariate shift in RKHS-based nonparametric regressioncited by 54
  2. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 42
  3. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 42
  4. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 38
  5. 2022A new similarity measure for covariate shift with applications to nonparametric regressioncited by 33
  6. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  7. 2017A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)cited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 7; 2026: 7.

26 papers.

Sort:

By area:

  1. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  2. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
  3. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 30
  4. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  5. 2023Pseudo-Labeling for Kernel Ridge Regression under Covariate Shiftcited by 17
  6. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  7. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 11
  8. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  9. 2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
  10. 2023Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuroncited by 8
  11. 2024On the Benefits of Over-parameterization for Out-of-Distribution Generalizationcited by 7
  12. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
14 more
  1. 2025Memory-Statistics Tradeoff in Continual Learning with Structural Regularizationcited by 5
  2. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  3. 2025Transfer Learning for Benign Overfitting in High-Dimensional Linear Regressioncited by 3
  4. 2023Out of the Ordinary: Spectrally Adapting Regression for Covariate Shiftcited by 2
  5. 2026Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methodscited by 2
  6. 2025Gradient Descent with Provably Tuned Learning-rate Schedulescited by 1
  7. 2024DeepMachining: Online Prediction of Machining Errors of Lathe Machinescited by 0
  8. 2024Memorization Capacity for Additive Fine-Tuning with Small ReLU Networkscited by 0
  9. 2024Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shiftcited by 0
  10. 2026Theory of Continual Learning Against Data Poisoning Attackscited by 0
  11. 2026Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descentcited by 0
  12. 2026Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networkscited by 0
  13. 2026Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networkscited by 0
  14. 2026Feature Normalization under Finite Training Budgetscited by 0