DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
Knowledge distillation (KD) frameworks place teacher and student on the same GPUs, so per-GPU memory limits the teachers that fit. DisaggKD separates them across nodes: teacher replicas stream hidden states to the student, a KD loss over row windows and vocabulary shards equals the full-vocabulary loss in exact arithmetic without forming the full-vocabulary logits, and a prefetch ring paces producer and trainer. Closed-form memory and traffic models locate the memory limits of each layout. On the Qwen3.5-35B-A3B to 9B pair, the disaggregated layout matches the LM loss of the colocated run with the same loss and is faster than that run; our fused loss also lets a colocated run train on as few nodes at faster speed, while the split is memory efficient when the student spans several nodes. On the larger pairs, colocated runs with a fused loss still run OOM where the split trains, because teacher weights and student states fill the budget.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 20 of its 29 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
20 of 29 references.
Closest ancestors · 1
Same direction, newest first
- 2025Distillation Scaling Laws12 Feb
Other references · 19
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
- 2018Online normalizer calculation for softmaxcited by 162
- 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
7 more
- 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
- 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
- 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 40
- 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
- 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27
- 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
- 2026Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Losscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024All You Need in Knowledge Distillation Is a Tailored Coordinate System
- 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
- 2024Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Loss
- 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
- 2025UNDO: Understanding Distillation as Optimization
- 2022Rethinking Knowledge Distillation via Cross-Entropy
- 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
- 2024Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias