mlsanityData to 14 Oct 2026

Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers

arXiv 2010.11750 ↗First posted 22 Oct 2020stat.ML cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

The problem of learning one task using samples from another task is central to transfer learning. In this paper, we focus on answering the following question: when does combining the samples from two related tasks perform better than learning with one target task alone? This question is motivated by an empirical phenomenon known as negative transfer, which has been observed in practice. While the transfer effect from one task to another depends on factors such as their sample sizes and the spectrum of their covariance matrices, precisely quantifying this dependence has remained a challenging problem. In order to compare a transfer learning estimator to single-task learning, one needs to compare the risks between the two estimators precisely. Further, the comparison depends on the distribution shifts between the two tasks. This paper applies recent developments of random matrix theory to tackle this challenge in a high-dimensional linear regression setting with two tasks. We show precise high-dimensional asymptotics for the bias and variance of a classical hard parameter sharing (HPS) estimator in the proportional limit, where the sample sizes of both tasks increase proportionally with dimension at fixed ratios. The precise asymptotics apply to various types of distribution shifts, including covariate shifts, model shifts, and combinations of both. We illustrate these results in a random-effects model to mathematically prove a phase transition from positive to negative transfer as the number of source task samples increases. One insight from the analysis is that a rebalanced HPS estimator, which downsizes the source task when the model shift is high, achieves the minimax optimal rate. The finding regarding phase transition also applies to multiple tasks when covariates are shared across tasks. Simulations validate the accuracy of the high-dimensional asymptotics for finite dimensions.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 3; 2024: 5; 2025: 9; 2026: 8.

27 papers.

Sort:

By area:

  1. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  2. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  3. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 22
  4. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  5. 2023Pseudo-Labeling for Kernel Ridge Regression under Covariate Shiftcited by 17
  6. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  7. 2023Graph Neural Networks for Road Safety Modeling: Datasets and Evaluations for Accident Analysiscited by 13
  8. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  9. 2025Efficient Ensemble for Fine-tuning Language Models on Multiple Datasetscited by 10
  10. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  11. 2024Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecastingcited by 8
  12. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimationcited by 8
15 more
  1. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 7
  2. 2025Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingnesscited by 5
  3. 2026Efficient Estimation of Kernel Surrogate Models for Task Attributioncited by 4
  4. 2024Collaborative Learning with Different Labeling Functionscited by 2
  5. 2025Optimal Regularization for Performative Learningcited by 1
  6. 2025Efficiently Learning Branching Networks for Multitask Algorithmic Reasoningcited by 1
  7. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 1
  8. 2025Source-Optimal Training is Transfer-Suboptimalcited by 0
  9. 2025Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimationcited by 0
  10. 2026On the Power of Source Screening for Learning Shared Feature Extractorscited by 0
  11. 2026Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networkscited by 0
  12. 2026Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingnesscited by 0
  13. 2026MoPLEx: Estimating Plackett-Luce Mixture Models for Multi-Objective Alignmentcited by 0
  14. 2026Long-Context Demonstration Selection Using State Space Modelscited by 0
  15. 2026High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learningcited by 0