mlsanityData to 7 Oct 2026

Churn Reduction via Distillation

arXiv 2106.02654 ↗First posted 4 Jun 2021cs.LG cs.AI stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i.e. predictive "churn"). If model retraining results in vastly different behavior, then it could cause negative effects in downstream systems, especially if this churn can be avoided with limited impact on model accuracy. In this paper, we show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines on a wide range of datasets and model architectures, including fully-connected networks, convolutional networks, and transformers.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

24 of 41 references.

References · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  5. 2013Do Deep Nets Really Need to be Deep?cited by 466
  6. 2015Policy Distillationcited by 414
  7. 2018How Does Batch Normalization Help Optimization?cited by 355
  8. 2018Model compression via distillation and quantizationcited by 287
  9. 2015Unifying distillation and privileged informationcited by 218
  10. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  11. 2017Learning from Noisy Labels with Distillationcited by 160
  12. 2021Towards Understanding Knowledge Distillationcited by 158
12 more
  1. 2018Large scale distributed neural network training through online distillationcited by 144
  2. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  3. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  4. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  5. 2016Satisfying Real-world Goals with Dataset Constraintscited by 68
  6. 2017Visual Relationship Detection with Internal and External Linguistic Knowledge Distillationcited by 60
  7. 2018Two-Player Games for Efficient Non-Convex Constrained Optimizationcited by 55
  8. 2021On the Reproducibility of Neural Network Predictionscited by 28
  9. 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 27
  10. 2019Hypothesis Set Stability and Generalizationcited by 23
  11. 2020Why distillation helps: a statistical perspectivecited by 21
  12. 2021Locally Adaptive Label Smoothing for Predictive Churncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 3; 2024: 4; 2025: 2; 2026: 3.

17 papers.

Sort:

  1. 2021Assessing Generalization of SGD via Disagreementcited by 116
  2. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  3. 2024Predictive Churn with the Set of Good Modelscited by 15
  4. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  5. 2022ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Trainingcited by 7
  6. 2022Predicting on the Edge: Identifying Where a Larger Model Does Bettercited by 6
  7. 2022Reducing Model Jitter: Stable Re-training of Semantic Parsers in Production Environmentscited by 6
  8. 2022On the Prediction Instability of Graph Neural Networkscited by 6
  9. 2025Mitigating Negative Flips via Margin Preserving Trainingcited by 3
  10. 2024FINEST: Stabilizing Recommendations by Rank-Preserving Fine-Tuningcited by 2
  11. 2023Maintaining Stability and Plasticity for Predictive Churn Reductioncited by 1
  12. 2023Measuring and Mitigating Local Instability in Deep Neural Networkscited by 1
5 more
  1. 2026Reducing cross-sample prediction churn in scientific machine learningcited by 1
  2. 2026A Stationary (and Therefore Compatible) Representation is All You Needcited by 1
  3. 2023Distilling Influences to Mitigate Prediction Churn in Graph Neural Networkscited by 0
  4. 2025Fractal basin geometry and the limits to predictability and reproducibility in deep learningcited by 0
  5. 2026Learn the Solid, Not the File: Canonical Inputs for Neural Networks on CAD Boundary Representationscited by 0