mlsanityData to 14 Oct 2026

Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models

arXiv 2502.16849 ↗First posted 24 Feb 2025stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Unsupervised pre-training and transfer learning are commonly used techniques to initialize training algorithms for neural networks, particularly in settings with limited labeled data. In this paper, we study the effects of unsupervised pre-training and transfer learning on the sample complexity of high-dimensional supervised learning. Specifically, we consider the problem of training a single-layer neural network via online stochastic gradient descent. We establish that pre-training and transfer learning (under concept shift) reduce sample complexity by polynomial factors (in the dimension) under very general assumptions. We also uncover some surprising settings where pre-training grants exponential improvement over random initialization in terms of sample complexity.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 42 of its 67 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

42 of 67 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Mask R-CNNcited by 7,277
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  6. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 612
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2016Learning to reinforcement learncited by 377
  9. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  10. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  11. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  12. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
30 more
  1. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 125
  2. 2020Domain Adaptation with Conditional Distribution Matching and Generalized Label Shiftcited by 125
  3. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  4. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  5. 2019Generalizing to unseen domains via distribution matchingcited by 110
  6. 2016A Geometric Analysis of Phase Retrievalcited by 92
  7. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  8. 2020On the Value of Target Data in Transfer Learningcited by 65
  9. 2019Domain Adaptation with Asymmetrically-Relaxed Distribution Alignmentcited by 61
  10. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  11. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  12. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  13. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  14. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  15. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 44
  16. 2020Phase retrieval in high dimensions: Statistical and computational phase transitionscited by 39
  17. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  18. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  19. 2017Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimationcited by 26
  20. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 26
  21. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 25
  22. 2020Minimax optimal approaches to the label shift problem in non-parametric settingscited by 20
  23. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  24. 2021On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependenciescited by 18
  25. 2023On Single Index Models beyond Gaussian Datacited by 17
  26. 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
  27. 2021Analysis of feature learning in weight-tied autoencoders via the mean field lenscited by 14
  28. 2019Transfer Learning based Detection of Diabetic Retinopathy from Small Datasetcited by 7
  29. 2024Learning Orthogonal Multi-Index Models: A Fine-Grained Information Exponent Analysiscited by 6
  30. 2024Semi-Supervised Sparse Gaussian Classification: Provable Benefits of Unlabeled Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  2. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 1
  3. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 1
  4. 2026Sample Complexity of Transfer Learning: An Optimal Transport Approachcited by 0
  5. 2026Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networkscited by 0
  6. 2026Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networkscited by 0