# Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

arXiv: https://arxiv.org/abs/2205.10490 · HTML: https://mlsanity.com/paper/2205.10490/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 39 of its 60 references that are in the map.
- First posted on arXiv 21 May 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Knowledge Distillation from Internal Representations](https://mlsanity.com/paper/1910.03723/) (2019)
- [Heterogeneous Knowledge Distillation using Information Flow Modeling](https://mlsanity.com/paper/2005.00727/) (2020)
- [ALP-KD: Attention-Based Layer Projection for Knowledge Distillation](https://mlsanity.com/paper/2012.14022/) (2020)
- [MEAL: Multi-Model Ensemble via Adversarial Learning](https://mlsanity.com/paper/1812.02425/) (2018)

## Built on by (most cited first, 3 of 3)

- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Decoupled Entropy Minimization](https://mlsanity.com/paper/2511.03256/) (2025)
- [Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation](https://mlsanity.com/paper/2606.24557/) (2026)

## Abstract (arXiv)

Black-Box Knowledge Distillation (B2KD) is a formulated problem for cloud-to-edge model compression with invisible data and models hosted on the server. B2KD faces challenges such as limited Internet exchange and edge-cloud disparity of data distributions. In this paper, we formalize a two-step workflow consisting of deprivatization and distillation, and theoretically provide a new optimization direction from logits to cell boundary different from direct logits alignment. With its guidance, we propose a new method Mapping-Emulation KD (MEKD) that distills a black-box cumbersome model into a lightweight one. Our method does not differentiate between treating soft or hard responses, and consists of: 1) deprivatization: emulating the inverse mapping of the teacher function with a generator, and 2) distillation: aligning low-dimensional logits of the teacher and student models by reducing the distance of high-dimensional image points. For different teacher-student pairs, our method yields inspiring distillation performance on various benchmarks, and outperforms the previous state-of-the-art approaches.