mlsanityData to 13 Oct 2026

High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws

arXiv 2410.18837 ↗First posted 24 Oct 2024stat.ML cs.LGcited by 22

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

A growing number of machine learning scenarios rely on knowledge distillation where one uses the output of a surrogate model as labels to supervise the training of a target model. In this work, we provide a sharp characterization of this process for ridgeless, high-dimensional regression, under two settings: (i) model shift, where the surrogate model is arbitrary, and (ii) distribution shift, where the surrogate model is the solution of empirical risk minimization with out-of-distribution data. In both cases, we characterize the precise risk of the target model through non-asymptotic bounds in terms of sample size and data distribution under mild conditions. As a consequence, we identify the form of the optimal surrogate model, which reveals the benefits and limitations of discarding weak features in a data-dependent fashion. In the context of weak-to-strong (W2S) generalization, this has the interpretation that (i) W2S training, with the surrogate as the weak model, can provably outperform training with strong labels under the same data budget, but (ii) it is unable to improve the data scaling law. We validate our results on numerical experiments both on ridgeless regression and on neural network architectures.

Key facts. Cited by 22 papers in the map (data through 13 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 42 of its 55 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

42 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023On student-teacher deviations in distillation: does it pay to disobey?30 Jan
  2. 2023Supervision Complexity and its Role in Knowledge Distillation28 Jan
  3. 2021Learning curves of generic features maps for realistic datasets with a teacher-student model16 Feb

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  3. 2020Scaling Laws for Neural Language Modelscited by 6,494
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,908
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 920
  7. 2019Benign Overfitting in Linear Regressioncited by 545
  8. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 523
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 510
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 454
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 411
  12. 2021Explaining Neural Scaling Lawscited by 355
27 more
  1. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  2. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 172
  3. 2020Benign overfitting in ridge regressioncited by 152
  4. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  5. 2022A Solvable Model of Neural Scaling Lawscited by 107
  6. 2024A Dynamical Model of Neural Scaling Lawscited by 107
  7. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  8. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  9. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 88
  10. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  11. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 67
  12. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  13. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  14. 2022Dimension free ridge regressioncited by 50
  15. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 47
  16. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  17. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 40
  18. 2019The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regimecited by 27
  19. 2020Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transferscited by 27
  20. 2023Towards a statistical theory of data selection under weak supervisioncited by 27
  21. 2024Scaling laws for learning with real and surrogate datacited by 27
  22. 2020Why distillation helps: a statistical perspectivecited by 21
  23. 2021Self-training Converts Weak Learners to Strong Learners in Mixture Modelscited by 19
  24. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 19
  25. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 18
  26. 2024Generalization error of min-norm interpolators in transfer learningcited by 15
  27. 2024Minimum-Norm Interpolation Under Covariate Shiftcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 8.

22 papers.

Sort:

By area:

  1. 2025Distillation Scaling Lawscited by 61
  2. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 14
  3. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 13
  4. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 11
  5. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 9
  6. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  7. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 7
  8. 2025Prediction-Powered Semi-Supervised Learning with Online Power Tuningcited by 5
  9. 2025Weak-to-Strong Generalization under Distribution Shiftscited by 4
  10. 2025Provably Improving Generalization of Few-Shot Models with Synthetic Datacited by 3
  11. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 3
  12. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 2
10 more
  1. 2025Optimal Regularization for Performative Learningcited by 1
  2. 2026SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generationcited by 1
  3. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  4. 2025Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learningcited by 0
  5. 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgettingcited by 0
  6. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 0
  7. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 0
  8. 2026What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regressioncited by 0
  9. 2026Prediction-Only Distillation in Linear and Logistic Regressioncited by 0
  10. 2026Scaling Properties of Same-Family On-Policy Distillationcited by 0