# Pretraining and the Lasso

arXiv: https://arxiv.org/abs/2401.12911 · HTML: https://mlsanity.com/paper/2401.12911/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2024.

## Built on by (most cited first, 8 of 8)

- [Representation Retrieval Learning for Heterogeneous Data Integration](https://mlsanity.com/paper/2503.09494/) (2025)
- [TL-PCA: Transfer Learning of Principal Component Analysis](https://mlsanity.com/paper/2410.10805/) (2024)
- [Using Pre-training and Interaction Modeling for ancestry-specific disease prediction in UK Biobank](https://mlsanity.com/paper/2404.17626/) (2024)
- [Transfer Learning in ℓ1 Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis](https://mlsanity.com/paper/2409.17704/) (2024)
- [Statistical Learning for Heterogeneous Treatment Effects: Pretraining, Prognosis, and Prediction](https://mlsanity.com/paper/2505.00310/) (2025)
- [A Stable Lasso](https://mlsanity.com/paper/2511.02306/) (2025)
- [Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing](https://mlsanity.com/paper/2602.16531/) (2026)
- [Transfer Learning in Nonparametric Regression with Deep ReLU Networks](https://mlsanity.com/paper/2608.20255/) (2026)

## Abstract (arXiv)

Pretraining is a popular and powerful paradigm in machine learning to pass information from one model to another. As an example, suppose one has a modest-sized dataset of images of cats and dogs, and plans to fit a deep neural network to classify them from the pixel features. With pretraining, we start with a neural network trained on a large corpus of images, consisting of not just cats and dogs but hundreds of other image types. Then we fix all of the network weights except for the top layer(s) (which makes the final classification) and train (or "fine tune") those weights on our dataset. This often results in dramatically better performance than the network trained solely on our smaller dataset. In this paper, we ask the question "Can pretraining help the lasso?". We develop a framework for the lasso in which a model is fit to a large dataset, and then fine-tuned using a smaller dataset. This latter dataset can be a subset of the original dataset, or it can be a dataset with a different but related outcome. This framework has a wide variety of applications, including stratified models, multinomial responses, multi-response models, conditional average treatment estimation and even gradient boosting. In the stratified model setting, the pretrained lasso pipeline estimates the coefficients common to all groups at the first stage, and then estimates the group-specific coefficients at the second "fine-tuning" stage. We show that under appropriate assumptions, the support recovery rate of the common coefficients is superior to that of the usual lasso trained only on individual groups. This separate identification of common and individual coefficients can also be useful for scientific understanding.