# Transfer Learning in ℓ1 Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis

arXiv: https://arxiv.org/abs/2409.17704 · HTML: https://mlsanity.com/paper/2409.17704/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 65 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017)
- [Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimality](https://mlsanity.com/paper/2006.10593/) (2020)
- [Transfer Learning under High-dimensional Generalized Linear Models](https://mlsanity.com/paper/2105.14328/) (2021)
- [A framework to characterize performance of LASSO algorithms](https://mlsanity.com/paper/1303.7291/) (2013)
- [Transfer Learning in Large-scale Gaussian Graphical Models with False Discovery Rate Control](https://mlsanity.com/paper/2010.11037/) (2020)
- [Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)](https://mlsanity.com/paper/2006.06581/) (2020)
- [Solvable Model for Inheriting the Regularization through Knowledge Distillation](https://mlsanity.com/paper/2012.00194/) (2020)
- [High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality](https://mlsanity.com/paper/2309.16476/) (2023)
- [Bayesian multi-domain learning for cancer subtype discovery from next-generation sequencing count data](https://mlsanity.com/paper/1810.09433/) (2018)
- [Pretraining and the Lasso](https://mlsanity.com/paper/2401.12911/) (2024)
- [Derivatives and residual distribution of regularized M-estimators with application to adaptive tuning](https://mlsanity.com/paper/2107.05143/) (2021)
- [The Role of Pseudo-labels in Self-training Linear Classifiers on High-dimensional Gaussian Mixture Data](https://mlsanity.com/paper/2205.07739/) (2022)
- [Regression modeling on stratified data with the lasso](https://mlsanity.com/paper/1508.05476/) (2015)
- [Average case analysis of Lasso under ultra-sparse conditions](https://mlsanity.com/paper/2302.13093/) (2023)
- [Precise Asymptotics of Bagging Regularized M-estimators](https://mlsanity.com/paper/2409.15252/) (2024)
- [Semi-Analytic Resampling in Lasso](https://mlsanity.com/paper/1802.10254/) (2018)
- [A statistical mechanics approach to de-biasing and uncertainty estimation in LASSO for random measurements](https://mlsanity.com/paper/1803.09927/) (2018)
- [Adaptive Sparse Estimation with Side Information](https://mlsanity.com/paper/1811.11930/) (2018)
- [Asymptotic Dynamics of Alternating Minimization for Bilinear Regression](https://mlsanity.com/paper/2402.04751/) (2024)

## Abstract (arXiv)

Transfer learning techniques aim to leverage information from multiple related datasets to enhance prediction quality against a target dataset. Such methods have been adopted in the context of high-dimensional sparse regression, and some Lasso-based algorithms have been invented: Trans-Lasso and Pretraining Lasso are such examples. These algorithms require the statistician to select hyperparameters that control the extent and type of information transfer from related datasets. However, selection strategies for these hyperparameters, as well as the impact of these choices on the algorithm's performance, have been largely unexplored. To address this, we conduct a thorough, precise study of the algorithm in a high-dimensional setting via an asymptotic analysis using the replica method. Our approach reveals a surprisingly simple behavior of the algorithm: Ignoring one of the two types of information transferred to the fine-tuning stage has little effect on generalization performance, implying that efforts for hyperparameter selection can be significantly reduced. Our theoretical findings are also empirically supported by applications on real-world and semi-artificial datasets using the IMDb and MNIST datasets, respectively.