mlsanityData to 13 Oct 2026

Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

arXiv 2608.27729 ↗First posted 27 Aug 2026cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students are attractive but knowledge distillation gains are typically reported single-seed, at scales where seed variance is unknown. On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy, using three to six seeds for key configurations. We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points; (ii) three of seven KD variants exhibit bimodal collapse, with at least one in three to five seeds falling below 55% accuracy while the others train normally, and a fourth showing elevated variance; (iii) collapse has distinct modes -- wrong-function selection for ce_kd and ce_paraphrase, and a previously undocumented output-truncation mode for reasoning_kd, where the model emits reasoning but terminates before producing a function name (0.9% accuracy); (iv) only progressive_kd and rank_kd avoid collapse across observed seeds, with sigma <= 3.9 pp; (v) a naive cross-split +3.78 pp gain from input enrichment reverses to -2.70 pp under controlled within-split multi-seed re-testing. Single-seed evaluation is therefore unable to detect central failure modes in small-model KD.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 16 of its 20 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

16 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Selective Knowledge Distillation for Neural Machine Translation27 May

Other references · 15

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,904
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,313
  6. 2016Sequence-Level Knowledge Distillationcited by 770
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 719
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 381
  9. 2022Teaching Small Language Models to Reasoncited by 282
  10. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  11. 2021Does Knowledge Distillation Really Work?cited by 188
  12. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 71
3 more
  1. 2021Annealing Knowledge Distillationcited by 61
  2. 2024TinyAgent: Function Calling at the Edgecited by 55
  3. 2024Octopus v2: On-device language model for super agentcited by 45

Built on by

Not cited yet.