mlsanityData to 14 Oct 2026

Provable Weak-to-Strong Generalization via Benign Overfitting

arXiv 2410.04638 ↗First posted 6 Oct 2024cs.LG stat.MLcited by 16

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

The classic teacher-student model in machine learning posits that a strong teacher supervises a weak student to improve the student's capabilities. We instead consider the inverted situation, where a weak teacher supervises a strong student with imperfect pseudolabels. This paradigm was recently brought forth by Burns et al.'23 and termed weak-to-strong generalization. We theoretically investigate weak-to-strong generalization for binary and multilabel classification in a stylized overparameterized spiked covariance model with Gaussian covariates where the weak teacher's pseudolabels are asymptotically like random guessing. Under these assumptions, we provably identify two asymptotic phases of the strong student's generalization after weak supervision: (1) successful generalization and (2) random guessing. Our techniques should eventually extend to weak-to-strong multiclass classification. Towards doing so, we prove a tight lower tail inequality for the maximum of correlated Gaussians, which may be of independent interest. Understanding the multilabel setting reinforces the value of using logits for weak supervision when they are available.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 54 of its 75 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

54 of 75 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Knowledge Distillation Performs Partial Variance Reduction27 May
  2. 2023Random Teachers are Good Teachers23 Feb
  3. 2023Learning From Biased Soft Labels16 Feb
  4. 2023Understanding Self-Distillation in the Presence of Label Noise30 Jan
  5. 2023On student-teacher deviations in distillation: does it pay to disobey?30 Jan
  6. 2022What Knowledge Gets Distilled in Knowledge Distillation?31 May
  7. 2022Learning curves for the multi-class teacher-student perceptron22 Mar
  8. 2021Does Knowledge Distillation Really Work?10 Jun
  9. 2021Towards Understanding Knowledge Distillation27 May

Other references · 45

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2020Knowledge Distillation: A Surveycited by 1,603
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
33 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  4. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  8. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  9. 2019Two models of double descent for weak featurescited by 195
  10. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  11. 2019Harmless interpolation of noisy data in regressioncited by 121
  12. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  13. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  14. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  15. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  16. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  17. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  18. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  19. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  20. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  21. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  22. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  23. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  24. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  25. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  26. 2024Transcendence: Generative Models Can Outperform The Experts That Train Themcited by 20
  27. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  28. 2022Generalization for multiclass classification with overparameterized linear modelscited by 8
  29. 2023General Loss Functions Lead to (Approximate) Interpolation in High Dimensionscited by 4
  30. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  31. 2015Asymptotics of Empirical Eigen-structure for Ultra-high Dimensional Spiked Covariance Modelcited by 3
  32. 2023New Equivalences Between Interpolation and SVMs: Kernels and Structured Featurescited by 3
  33. 2023Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 15
  2. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 12
  3. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 10
  4. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  5. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 4
  6. 2025Weak-to-Strong Generalization under Distribution Shiftscited by 4
  7. 2025How to Mitigate Overfitting in Weak-to-strong Generalization?cited by 3
  8. 2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
  9. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 2
  10. 2026Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)cited by 2
  11. 2025Limitations of refinement methods for weak to strong generalizationcited by 1
  12. 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgettingcited by 1
4 more
  1. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 1
  2. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  3. 2026Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teachercited by 0
  4. 2026What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Selective Weak-to-Strong Generalization
  2. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws
  3. 2024Weak-to-Strong Generalization Through the Data-Centric Lens
  4. 2024Sample-Efficient Agnostic Boosting
  5. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
  6. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
  7. 2025Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalization
  8. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation