Aligning Logits Generatively for Principled Black-Box Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
Black-Box Knowledge Distillation (B2KD) is a formulated problem for cloud-to-edge model compression with invisible data and models hosted on the server. B2KD faces challenges such as limited Internet exchange and edge-cloud disparity of data distributions. In this paper, we formalize a two-step workflow consisting of deprivatization and distillation, and theoretically provide a new optimization direction from logits to cell boundary different from direct logits alignment. With its guidance, we propose a new method Mapping-Emulation KD (MEKD) that distills a black-box cumbersome model into a lightweight one. Our method does not differentiate between treating soft or hard responses, and consists of: 1) deprivatization: emulating the inverse mapping of the teacher function with a generator, and 2) distillation: aligning low-dimensional logits of the teacher and student models by reducing the distance of high-dimensional image points. For different teacher-student pairs, our method yields inspiring distillation performance on various benchmarks, and outperforms the previous state-of-the-art approaches.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 21 May 2022.
Builds on
39 of 60 references.
References · 39
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2015Domain-Adversarial Training of Neural Networkscited by 3,023
- 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
- 2017Improved Training of Wasserstein GANscited by 2,223
- 2014Conditional Generative Adversarial Netscited by 2,152
- 2020Knowledge Distillation: A Surveycited by 1,578
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
27 more
- 2016Stealing Machine Learning Models via Prediction APIscited by 602
- 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
- 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
- 2013Do Deep Nets Really Need to be Deep?cited by 466
- 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
- 2022Decoupled Knowledge Distillationcited by 380
- 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
- 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 285
- 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
- 2019Data-Free Learning of Student Networkscited by 203
- 2021Does Knowledge Distillation Really Work?cited by 187
- 2017Learning Discrete Representations via Information Maximizing Self-Augmented Trainingcited by 149
- 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
- 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
- 2019Knowledge Distillation from Internal Representationscited by 87
- 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
- 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
- 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
- 2018Learning Student Networks via Feature Embeddingcited by 43
- 2018Geometric Understanding of Deep Learningcited by 41
- 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
- 2019Conditional Teacher-Student Learningcited by 32
- 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
- 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
- 2017A Geometric View of Optimal Transportation and Generative Modelcited by 31
- 2020Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Modelcited by 28
- 2018Adversarial Network Compressioncited by 9
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.