# On the Generalization for Transfer Learning: An Information-Theoretic Analysis

arXiv: https://arxiv.org/abs/2207.05377 · HTML: https://mlsanity.com/paper/2207.05377/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 94 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 30 of 37)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient](https://mlsanity.com/paper/1703.00102/) (2017)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [Reasoning About Generalization via Conditional Mutual Information](https://mlsanity.com/paper/2001.09122/) (2020)
- [Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates](https://mlsanity.com/paper/1911.02151/) (2019)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [Preserving Statistical Validity in Adaptive Data Analysis](https://mlsanity.com/paper/1411.2664/) (2014)
- [Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms](https://mlsanity.com/paper/2004.12983/) (2020)
- [Generalization Error Bounds for Noisy, Iterative Algorithms](https://mlsanity.com/paper/1801.04295/) (2018)
- [On the Value of Target Data in Transfer Learning](https://mlsanity.com/paper/2002.04747/) (2020)
- [A scoping review of transfer learning research on medical image analysis using ImageNet](https://mlsanity.com/paper/2004.13175/) (2020)
- [Information-theoretic generalization bounds for black-box learning algorithms](https://mlsanity.com/paper/2110.01584/) (2021)
- [Fast rates in statistical and online learning](https://mlsanity.com/paper/1507.02592/) (2015)
- [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1809.00357/) (2018)
- [Tighter expected generalization error bounds via Wasserstein distance](https://mlsanity.com/paper/2101.09315/) (2021)
- [Global optimization of Lipschitz functions](https://mlsanity.com/paper/1703.02628/) (2017)
- [PAC-Bayes Un-Expected Bernstein Inequality](https://mlsanity.com/paper/1905.13367/) (2019)
- [The Impact of Regularization on High-dimensional Logistic Regression](https://mlsanity.com/paper/1906.03761/) (2019)
- [KL Guided Domain Adaptation](https://mlsanity.com/paper/2106.07780/) (2021)
- [PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classes](https://mlsanity.com/paper/2106.09683/) (2021)
- [Generalization Bounds for Domain Adaptation](https://mlsanity.com/paper/1304.1574/) (2013)
- [Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayes](https://mlsanity.com/paper/1605.00252/) (2016)
- [Refined Error Bounds for Several Learning Algorithms](https://mlsanity.com/paper/1512.07146/) (2015)
- [TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning](https://mlsanity.com/paper/2103.01542/) (2021)
- [Information-theoretic analysis for transfer learning](https://mlsanity.com/paper/2005.08697/) (2020)
- [An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift](https://mlsanity.com/paper/2202.12123/) (2022)
- [Distant Domain Transfer Learning for Medical Imaging](https://mlsanity.com/paper/2012.06346/) (2020)

## Built on by (most cited first, 11 of 11)

- [Asymmetry of the Relative Entropy in the Regularization of Empirical Risk Minimization](https://mlsanity.com/paper/2410.02833/) (2024)
- [Understanding Knowledge Transferability for Transfer Learning: A Survey](https://mlsanity.com/paper/2507.03175/) (2025)
- [Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective](https://mlsanity.com/paper/2506.07861/) (2025)
- [Subgroups Matter for Robust Bias Mitigation](https://mlsanity.com/paper/2505.21363/) (2025)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- ['No' Matters: Out-of-Distribution Detection in Multimodality Long Dialogue](https://mlsanity.com/paper/2410.23883/) (2024)
- [Provable Performance Bounds for Digital Twin-driven Deep Reinforcement Learning in Wireless Networks: A Novel Digital-Twin Bisimulation Metric](https://mlsanity.com/paper/2502.17983/) (2025)
- [Adaptive prediction theory combining offline and online learning](https://mlsanity.com/paper/2512.00342/) (2025)
- [Empirical Risk Minimization with f-Divergence Regularization](https://mlsanity.com/paper/2601.13191/) (2026)
- [Gradient Transformer: Learning to Generate Updates for LLMs](https://mlsanity.com/paper/2605.27591/) (2026)
- [Generalization Theory for Through-the-Wall Radar Human Activity Recognition](https://mlsanity.com/paper/2607.08144/) (2026)

## Abstract (arXiv)

Transfer learning, or domain adaptation, is concerned with machine learning problems in which training and testing data come from possibly different probability distributions. In this work, we give an information-theoretic analysis of the generalization error and excess risk of transfer learning algorithms. Our results suggest, perhaps as expected, that the Kullback-Leibler (KL) divergence D(μ|μ') plays an important role in the characterizations where μ and μ' denote the distribution of the training data and the testing data, respectively. Specifically, we provide generalization error and excess risk upper bounds for learning algorithms where data from both distributions are available in the training phase. Recognizing that the bounds could be sub-optimal in general, we provide improved excess risk upper bounds for a certain class of algorithms, including the empirical risk minimization (ERM) algorithm, by making stronger assumptions through the textitcentral condition. To demonstrate the usefulness of the bounds, we further extend the analysis to the Gibbs algorithm and the noisy stochastic gradient descent method. We then generalize the mutual information bound with other divergences such as φ-divergence and Wasserstein distance, which may lead to tighter bounds and can handle the case when μ is not absolutely continuous with respect to μ'. Several numerical results are provided to demonstrate our theoretical findings. Lastly, to address the problem that the bounds are often not directly applicable in practice due to the absence of the distributional knowledge of the data, we develop an algorithm (called InfoBoost) that dynamically adjusts the importance weights for both source and target data based on certain information measures. The empirical results show the effectiveness of the proposed algorithm.