Formal Bayesian Transfer Learning via the Total Risk Prior

Existing methods for transfer learning struggle to deal with situations where the source datasets are limited and not guaranteed to be well-aligned with the target dataset. A typical strategy is to use the empirical loss minimizer on the source data as a prior mean for the target parameters. We propose using a risk minimizer conditional on source parameters instead. This allows us to construct a joint posterior for all parameters from the source and target datasets. As a consequence, we benefit from full Bayesian uncertainty quantification and can perform model averaging via Gibbs sampling over indicator variables governing the inclusion of each source dataset. We show how a particular instantiation of our prior leads to a Bayesian Lasso in a transformed coordinate system and describe connections between the Maximum a Posteriori estimate associated with our approach and the Trans-Lasso method. We find that the MAP estimator MSE-dominates the Trans-Lasso in the normal means setting when there is no regularization on the source datasets. Finally, we perform numerical experiments finding that our approach provides superior predictive performance relative to frequentist and Bayesian baselines on a genetics application, particularly when the source data are limited.