mlsanityData to 14 Oct 2026

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

arXiv 2606.01292 ↗First posted 31 May 2026cs.LG cs.AIcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization. While existing studies offer isolated insights, a unified theoretical framework explaining the efficacy of KT across these disparate regimes remains lacking. In this work, we establish a unified spectral analysis of SGD dynamics in high-dimensional linear regression, elucidating the efficiency of KT across seemingly disparate regimes. We characterize KT efficiency through two distinct mechanisms: Spectral Horizon Expansion in KD, which enables the capture of statistically inaccessible high-frequency signals, and Spectral Denoising in W2S, where the student acts as a filter for optimization noise. Our framework unifies these phenomena, revealing that the efficacy of transfer is governed by the interplay between implicit regularization and heterogeneous spectral learning speeds over the spectrum.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 34 of its 46 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

34 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws24 Oct
  2. 2024Provable Weak-to-Strong Generalization via Benign Overfitting6 Oct
  3. 2023On student-teacher deviations in distillation: does it pay to disobey?30 Jan
  4. 2021Does Knowledge Distillation Really Work?10 Jun

Other references · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2020Designing Network Design Spacescited by 960
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  11. 2019Benign Overfitting in Linear Regressioncited by 549
  12. 2018Born Again Neural Networkscited by 546
18 more
  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  2. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  5. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  6. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  7. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  8. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  9. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  10. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 38
  11. 2017A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)cited by 26
  12. 2021Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrumscited by 15
  13. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 15
  14. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 12
  15. 2024The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimizationcited by 8
  16. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  17. 2023Risk Bounds of Accelerated SGD for Overparameterized Linear Regressioncited by 7
  18. 2025Learning Curves of Stochastic Gradient Descent in Kernel Regressioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation
  2. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
  3. 2024Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching
  4. 2023Comparative Knowledge Distillation
  5. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic View
  6. 2021Knowledge Distillation as Semiparametric Inference
  7. 2023Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning
  8. 2023Knowledge Distillation Performs Partial Variance Reduction