mlsanityData to 7 Oct 2026

Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

arXiv 2111.01635 ↗First posted 2 Nov 2021cs.LG cs.IT math.ITcited by 14

Not on the map. Why?

Abstract (arXiv)

We provide an information-theoretic analysis of the generalization ability of Gibbs-based transfer learning algorithms by focusing on two popular transfer learning approaches, α-weighted-ERM and two-stage-ERM. Our key result is an exact characterization of the generalization behaviour using the conditional symmetrized KL information between the output hypothesis and the target training samples given the source samples. Our results can also be applied to provide novel distribution-free generalization error upper bounds on these two aforementioned Gibbs algorithms. Our approach is versatile, as it also characterizes the generalization errors and excess risks of these two Gibbs algorithms in the asymptotic regime, where they converge to the α-weighted-ERM and two-stage-ERM, respectively. Based on our theoretical results, we show that the benefits of transfer learning can be viewed as a bias-variance trade-off, with the bias induced by the source distribution and the variance induced by the lack of target samples. We believe this viewpoint can guide the choice of transfer learning algorithms in practice.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 27 of its 56 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

27 of 56 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  3. 2014How transferable are features in deep neural networks?cited by 1,216
  4. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  5. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  6. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  7. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  8. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  9. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  10. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  11. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  12. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
15 more
  1. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  2. 2020On the Value of Target Data in Transfer Learningcited by 62
  3. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  4. 2020Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networkscited by 30
  5. 2015A New PAC-Bayesian Perspective on Domain Adaptationcited by 29
  6. 2013Generalization Bounds for Domain Adaptationcited by 27
  7. 2020Concentration Inequalities for Statistical Inferencecited by 24
  8. 2020Information-theoretic analysis for transfer learningcited by 23
  9. 2020Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithmscited by 19
  10. 2020A Theory of Multiple-Source Adaptation with Limited Target Labeled Datacited by 18
  11. 2019Distribution-Dependent Analysis of Gibbs-ERM Principlecited by 17
  12. 2014Fast Rates by Transferring from Auxiliary Hypothesescited by 16
  13. 2021Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithmscited by 12
  14. 2019Information-Theoretic Understanding of Population Risk Improvement with Model Compressioncited by 11
  15. 2021Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL informationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 4; 2025: 1; 2026: 1.

14 papers.

Sort:

  1. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  2. 2022The Value of Out-of-Distribution Datacited by 12
  3. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  4. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 10
  5. 2022Learning Algorithm Generalization Error Bounds via Auxiliary Distributionscited by 6
  6. 2024On f-Divergence Principled Domain Adaptation: An Improved Frameworkcited by 6
  7. 2022How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?cited by 5
  8. 2025Variations on the Expectation due to Changes in the Probability Measurecited by 3
  9. 2023On the Generalization Error of Meta Learning for the Gibbs Algorithmcited by 2
  10. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  11. 2024Understanding Transfer Learning via Mean-field Analysiscited by 1
  12. 2023An information-Theoretic Approach to Semi-supervised Transfer Learningcited by 0
2 more
  1. 2024Depth F1: Improving Evaluation of Cross-Domain Text Classification by Measuring Semantic Generalizabilitycited by 0
  2. 2026Gradient Transformer: Learning to Generate Updates for LLMscited by 0