mlsanityData to 14 Oct 2026

An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation

arXiv 2401.06356 ↗First posted 12 Jan 2024cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

We present a large-scale empirical study of how choices of configuration parameters affect performance in knowledge distillation (KD). An example of such a KD parameter is the measure of distance between the predictions of the teacher and the student, common choices for which include the mean squared error (MSE) and the KL-divergence. Although scattered efforts have been made to understand the differences between such options, the KD literature still lacks a systematic study on their general effect on student performance. We take an empirical approach to this question in this paper, seeking to find out the extent to which such choices influence student performance across 13 datasets from 4 NLP tasks and 3 student sizes. We quantify the cost of making sub-optimal choices and identify a single configuration that performs well across the board.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 19 of its 31 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

19 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Knowledge distillation: A good teacher is patient and consistent9 Jun
  2. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation19 May

Other references · 17

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  12. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
5 more
  1. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  2. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 43
  3. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  4. 2022Cross-Task Knowledge Distillation in Multi-Task Recommendationcited by 22
  5. 2022Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answeringcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Generalization vs Fidelity Paradox in Knowledge Distillation
  2. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
  3. 2023On student-teacher deviations in distillation: does it pay to disobey?
  4. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration
  5. 2023Comparative Knowledge Distillation
  6. 2025UNDO: Understanding Distillation as Optimization
  7. 2026Rethinking Selective Knowledge Distillation
  8. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence