# Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

arXiv: https://arxiv.org/abs/2602.17565 · HTML: https://mlsanity.com/paper/2602.17565/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- First posted on arXiv 19 Feb 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Built on by (most cited first, 2 of 2)

- [Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models](https://mlsanity.com/paper/2605.17778/) (2026)
- [Optimal Self-Distillation for Rectified Flow via Linear Probing](https://mlsanity.com/paper/2607.14947/) (2026)

## Abstract (arXiv)

Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth labels and the teacher's own predictions using the same architecture and training data. Although SD has been empirically shown to often improve generalization, its formal guarantees remain limited. We study SD for ridge regression in unconstrained setting in which the mixing weight xi may be outside the unit interval. Conditioned on the training data and without any distributional assumptions, we prove that for any squared prediction risk (including out-of-distribution), the optimally mixed student strictly improves upon the ridge teacher for every regularization level λ > 0 at which the teacher ridge risk R(λ) is nonstationary (i.e., R'(λ) neq 0). We obtain a closed-form expression for the optimal mixing weight xi^star(λ) for any value of λ and show that it obeys the sign rule: operatornamesign(xi^star(λ))=-operatornamesign(R'(λ)). In particular, xi^star(λ) can be negative, which is the case in over-regularized regimes. To quantify the risk improvement due to SD, we derive exact deterministic equivalents for the optimal SD risk in the proportional asymptotics regime (where the sample and feature sizes n and p both diverge but their aspect ratio p/n converges) under general anisotropic covariance and deterministic signals. Our asymptotic analysis extends standard second-order ridge deterministic equivalents to their fourth-order analogs using block linearization, which may be of independent interest. From a practical standpoint, we propose a consistent one-shot tuning method to estimate xi^star without grid search, sample splitting, or refitting. Experiments on real-world datasets and pretrained neural network features support our theory and the one-shot tuning method.