mlsanityData to 7 Oct 2026

Transfer Learning Beyond Bounded Density Ratios

arXiv 2403.11963 ↗First posted 18 Mar 2024cs.LG cs.DS math.STcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the fundamental problem of transfer learning where a learning algorithm collects data from some source distribution P but needs to perform well with respect to a different target distribution Q. A standard change of measure argument implies that transfer learning happens when the density ratio dQ/dP is bounded. Yet, prior thought-provoking works by Kpotufe and Martinet (COLT, 2018) and Hanneke and Kpotufe (NeurIPS, 2019) demonstrate cases where the ratio dQ/dP is unbounded, but transfer learning is possible. In this work, we focus on transfer learning over the class of low-degree polynomial estimators. Our main result is a general transfer inequality over the domain ℝn, proving that non-trivial transfer learning for low-degree polynomials is possible under very mild assumptions, going well beyond the classical assumption that dQ/dP is bounded. For instance, it always applies if Q is a log-concave measure and the inverse ratio dP/dQ is bounded. To demonstrate the applicability of our inequality, we obtain new results in the settings of: (1) the classical truncated regression setting, where dQ/dP equals infinity, and (2) the more recent out-of-distribution generalization setting for in-context learning linear functions with transformers. We also provide a discrete analogue of our transfer inequality on the Boolean Hypercube -1,1n, and study its connections with the recent problem of Generalization on the Unseen of Abbe, Bengio, Lotfi and Rizk (ICML, 2023). Our main conceptual contribution is that the maximum influence of the error of the estimator widehatf-f^* under Q, Imax(widehatf-f^*), acts as a sufficient condition for transferability; when Imax(widehatf-f^*) is appropriately bounded, transfer is possible over the Boolean domain.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 52 of its 91 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

52 of 91 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  4. 2020In Search of Lost Domain Generalizationcited by 1,042
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  9. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
38 more
  1. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  2. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  3. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  4. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  5. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  6. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  7. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  8. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  9. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  10. 2018Marginal Singularity, and the Benefits of Labels in Covariate-Shiftcited by 65
  11. 2019Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifiercited by 64
  12. 2020On the Value of Target Data in Transfer Learningcited by 62
  13. 2022Optimally tackling covariate shift in RKHS-based nonparametric regressioncited by 53
  14. 2020Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examplescited by 44
  15. 2020Transfer Learning in Large-scale Gaussian Graphical Models with False Discovery Rate Controlcited by 38
  16. 2021Early-stopped neural networks are consistentcited by 34
  17. 2022A new similarity measure for covariate shift with applications to nonparametric regressioncited by 33
  18. 2022A Data-Based Perspective on Transfer Learningcited by 32
  19. 2020Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networkscited by 30
  20. 2018Efficient Statistics, in High Dimensions, from Truncated Samplescited by 28
  21. 2023Testable Learning with Distribution Shiftcited by 25
  22. 2020Computationally and Statistically Efficient Truncated Regressioncited by 23
  23. 2023Optimal Learners for Realizable Regression: PAC Learning and Online Learningcited by 23
  24. 2019Efficient Truncated Statistics with Unknown Truncationcited by 22
  25. 2021Efficient Algorithms for Learning from Coarse Labelscited by 21
  26. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  27. 2021Efficient Learning with Arbitrary Covariate Shiftcited by 15
  28. 2020Efficient Parameter Estimation of Truncated Boolean Product Distributionscited by 13
  29. 2020Truncated Linear Regression in High Dimensionscited by 13
  30. 2022Efficient Truncated Linear Regression with Unknown Noise Variancecited by 10
  31. 2013Inverse Density as an Inverse Problem: The Fredholm Equation Approachcited by 9
  32. 2023Limits of Model Selection under Transfer Learningcited by 9
  33. 2023Testing Convex Truncationcited by 8
  34. 2023Learning Hard-Constrained Models with One Samplecited by 7
  35. 2024Detecting Low-Degree Truncationcited by 7
  36. 2022Perfect Sampling from Pairwise Comparisonscited by 6
  37. 2023Tackling Combinatorial Distribution Shift: A Matrix Completion Perspectivecited by 5
  38. 2013Sparse Covers for Sums of Indicatorscited by 2

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2024Tolerant Algorithms for Learning with Arbitrary Covariate Shiftcited by 15
  2. 2024Efficient Discrepancy Testing for Learning with Distribution Shiftcited by 9
  3. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  4. 2025The Power of Iterative Filtering for Supervised Learning with (Heavy) Contaminationcited by 7
  5. 2025Learning Neural Networks with Distribution Shift: Efficiently Certifiable Guaranteescited by 4
  6. 2025Smoothed Analysis of Learning from Positive Samplescited by 2