Generative Distribution Distillation

In this paper, we formulate the knowledge distillation (KD) as a conditional generative problem and propose the textitGenerative Distribution Distillation (GenDD) framework. A naive textitGenDD baseline encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. To address these issues, we introduce a textitSplit Tokenization strategy, achieving stable and effective unsupervised KD. Additionally, we develop the textitDistribution Contraction technique to integrate label supervision into the reconstruction objective. Our theoretical proof demonstrates that textitGenDD with textitDistribution Contraction serves as a gradient-level surrogate for multi-task learning, realizing efficient supervised training without explicit classification loss on multi-step sampling image representations. To evaluate the effectiveness of our method, we conduct experiments on balanced, imbalanced, and unlabeled data. Experimental results show that textitGenDD performs competitively in the unsupervised setting, significantly surpassing KL baseline by 16.29% on ImageNet validation set. With label supervision, our ResNet-50 achieves 82.28% top-1 accuracy on ImageNet in 600 epochs training, establishing a new state-of-the-art.