mlsanityData to 14 Oct 2026

Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimes

arXiv 2306.04901 ↗First posted 8 Jun 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Transfer learning is a useful technique for achieving improved performance and reducing training costs by leveraging the knowledge gained from source tasks and applying it to target tasks. Assessing the effectiveness of transfer learning relies on understanding the similarity between the ground truth of the source and target tasks. In real-world applications, tasks often exhibit partial similarity, where certain aspects are similar while others are different or irrelevant. To investigate the impact of partial similarity on transfer learning performance, we focus on a linear regression model with two distinct sets of features: a common part shared across tasks and a task-specific part. Our study explores various types of transfer learning, encompassing two options for parameter transfer. By establishing a theoretical characterization on the error of the learned model, we compare these transfer learning options, particularly examining how generalization performance changes with the number of features/parameters in both underparameterized and overparameterized regimes. Furthermore, we provide practical guidelines for determining the number of features in the common and task-specific parts for improved generalization performance. For example, when the total number of features in the source task's learning model is fixed, we show that it is more advantageous to allocate a greater number of redundant features to the task-specific part rather than the common part. Moreover, in specific scenarios, particularly those characterized by high noise levels and small true parameters, sacrificing certain true features in the common part in favor of employing more redundant features in the task-specific part can yield notable benefits.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

20 of 28 references.

References · 20

  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  5. 2019Two models of double descent for weak featurescited by 195
  6. 2018To understand deep learning we need to understand kernel learningcited by 187
  7. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  8. 2019Harmless interpolation of noisy data in regressioncited by 121
  9. 2020Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimalitycited by 90
  10. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  11. 2021Transfer Learning under High-dimensional Generalized Linear Modelscited by 70
  12. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 25
8 more
  1. 2019Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolationcited by 23
  2. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 22
  3. 2020Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Taskscited by 16
  4. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  5. 2021The Dynamics of Gradient Descent for Overparametrized Neural Networkscited by 11
  6. 2022A Class of Geometric Structures in Transfer Learning: Minimax Bounds and Optimalitycited by 9
  7. 2022On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Modelcited by 6
  8. 2023Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 1.

4 papers.

  1. 2025Transfer Learning for Benign Overfitting in High-Dimensional Linear Regressioncited by 3
  2. 2023HKTGNN: Hierarchical Knowledge Transferable Graph Neural Network-based Supply Chain Risk Assessmentcited by 0
  3. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  4. 2026Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networkscited by 0