mlsanityData to 14 Oct 2026

Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems

arXiv 2606.27797 ↗First posted 26 Jun 2026cs.DC cs.AI cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge Distillation (KD) enables training smaller student models under the guidance of larger teacher models, and the widely adopted TRL library implements it. Yet, TRL treats both models symmetrically, missing opportunities to exploit their pronounced asymmetry in memory footprint, and communication requirements. This paper presents an HPC-aware methodology for KD that decouples teacher and student partitioning efficiently. Our approach achieves up to 67% higher samples-per-second than TRL by avoiding unnecessary teacher-model data structures and selecting the best split strategy. We combine vertical and horizontal partitioning of models, deriving an analytical expression that identifies the existence of inflection points between splitting regimes. These results showed that exploiting teacher--student asymmetry through topology-aware parallelism notably accelerated GKD training on production HPC clusters at our company

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 10 of its 13 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

10 of 13 references.

References · 10

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2020torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillationcited by 19
  10. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalizing Teacher Networks for Effective Knowledge Distillation Across Student Architectures
  2. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
  3. 2025HPM-KD: Hierarchical Progressive Multi-Teacher Framework for Knowledge Distillation and Efficient Model Compression
  4. 2023Can a student Large Language Model perform as well as it's teacher?
  5. 2026Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
  6. 2023Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication
  7. 2023Comparative Knowledge Distillation
  8. 2026KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models