mlsanityData to 14 Oct 2026

Prediction-Only Distillation in Linear and Logistic Regression

arXiv 2607.15450 ↗First posted 16 Jul 2026math.ST cs.LG stat.MLcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 39 of its 68 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

39 of 68 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models18 May
  2. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model27 Jan
  3. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws24 Oct
  4. 2024Understanding the Gains from Repeated Self-Distillation5 Jul
  5. 2024Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels16 Feb
  6. 2023Understanding Self-Distillation in the Presence of Label Noise30 Jan
  7. 2022Knowledge Distillation with the Reused Teacher Classifier26 Mar

Other references · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  4. 2018Born Again Neural Networkscited by 546
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  8. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  9. 2020Meta Pseudo Labelscited by 374
  10. 2019Variational Information Distillation for Knowledge Transfercited by 324
  11. 2023Self-Consuming Generative Models Go MADcited by 250
  12. 2021Deep learning: a statistical viewpointcited by 240
20 more
  1. 2015Unifying distillation and privileged informationcited by 233
  2. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  3. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  4. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  5. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  6. 2024Strong Model Collapsecited by 65
  7. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  8. 2020Solvable Model for Inheriting the Regularization through Knowledge Distillationcited by 16
  9. 2024Generalization error of min-norm interpolators in transfer learningcited by 16
  10. 2024Minimum-Norm Interpolation Under Covariate Shiftcited by 14
  11. 2022Mitigating multiple descents: A model-agnostic framework for risk monotonizationcited by 10
  12. 2025Recursive Learning Without Collapse: A Weighting-Based Stabilization Frameworkcited by 10
  13. 2021Can we globally optimize cross-validation loss? Quasiconvexity in ridge regressioncited by 9
  14. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 9
  15. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  16. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 7
  17. 2024Retraining with Predicted Hard Labels Provably Increases Model Accuracycited by 5
  18. 2026Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regressioncited by 3
  19. 2025Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passingcited by 1
  20. 2026From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sourcescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
  2. 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting
  3. 2025Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets
  4. 2023Random Teachers are Good Teachers
  5. 2023Self-Distillation for Gaussian Process Regression and Classification
  6. 2026IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
  7. 2022Robust Distillation for Worst-class Performance
  8. 2022Revisiting Self-Distillation