mlsanityData to 14 Oct 2026

On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective

arXiv 2505.18346 ↗First posted 23 May 2025stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Weak-to-strong generalization, where a student model trained on imperfect labels generated by a weaker teacher nonetheless surpasses that teacher, has been widely observed but the mechanisms that enable it have remained poorly understood. In this paper, through a theoretical analysis of simple models, we uncover three core mechanisms that can drive this phenomenon. First, by analyzing ridge regression, we study the interplay between the teacher and student regularization and prove that a student can compensate for a teacher's under-regularization and achieve lower test error. We also analyze the role of the parameterization regime of the models. Second, by analyzing weighted ridge regression, we show that a student model with a regularization structure more aligned to the target, can outperform its teacher. Third, in a nonlinear multi-index setting, we demonstrate that a student can learn easy, task-specific features from the teacher while leveraging its own broader pre-training to learn hard-to-learn features that the teacher cannot capture.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 43 of its 74 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

43 of 74 references.

References · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  5. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  6. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  7. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  8. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  9. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  10. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  11. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  12. 2024Model Collapse Demystified: The Case of Regressioncited by 82
31 more
  1. 2022Universality of empirical risk minimizationcited by 75
  2. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  3. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 61
  4. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  5. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  6. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  7. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  8. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  9. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  10. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  11. 2024Weak-to-Strong Reasoningcited by 33
  12. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  13. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  14. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  15. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  16. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 22
  17. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  18. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 21
  19. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  20. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  21. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  22. 2024Understanding the Gains from Repeated Self-Distillationcited by 15
  23. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 15
  24. 2025On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learningcited by 12
  25. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 12
  26. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 10
  27. 2024Risk and cross validation in ridge regression with correlated samplescited by 8
  28. 2025Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Losscited by 8
  29. 2024Asymptotics of Linear Regression with Linearly Dependent Datacited by 5
  30. 2024Signal-Plus-Noise Decomposition of Nonlinear Spiked Random Matrix Modelscited by 2
  31. 2024Generalization for Least Squares Regression With Simple Spiked Covariancescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 4
  2. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 2
  3. 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgettingcited by 1
  4. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 1
  5. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  6. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  7. 2026What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regressioncited by 0
  8. 2026Prediction-Only Distillation in Linear and Logistic Regressioncited by 0