# Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model

arXiv: https://arxiv.org/abs/2003.13960 · HTML: https://mlsanity.com/paper/2003.13960/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2020.

## Built on by (most cited first, 28 of 28)

- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition](https://mlsanity.com/paper/2208.05768/) (2022)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2207.12106/) (2022)
- [CTIN: Robust Contextual Transformer Network for Inertial Navigation](https://mlsanity.com/paper/2112.02143/) (2021)
- [Dual Discriminator Adversarial Distillation for Data-free Model Compression](https://mlsanity.com/paper/2104.05382/) (2021)
- [Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack](https://mlsanity.com/paper/2104.05921/) (2021)
- [MeaeQ: Mount Model Extraction Attacks with Efficient Queries](https://mlsanity.com/paper/2310.14047/) (2023)
- [When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation](https://mlsanity.com/paper/2203.09391/) (2022)
- [Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2211.03946/) (2022)
- [Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2306.08909/) (2023)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [Isotonic Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2107.01412/) (2021)
- [Towards the Fundamental Limits of Knowledge Transfer over Finite Domains](https://mlsanity.com/paper/2310.07838/) (2023)
- [Comparative Knowledge Distillation](https://mlsanity.com/paper/2311.02253/) (2023)
- [Deep Epidemiological Modeling by Black-box Knowledge Distillation: An Accurate Deep Learning Model for COVID-19](https://mlsanity.com/paper/2101.10280/) (2021)
- [Aligning Logits Generatively for Principled Black-Box Knowledge Distillation](https://mlsanity.com/paper/2205.10490/) (2022)
- [MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition](https://mlsanity.com/paper/2310.18450/) (2023)
- [Effective Decision Boundary Learning for Class Incremental Learning](https://mlsanity.com/paper/2301.05180/) (2023)
- [A Scalable and Efficient Iterative Method for Copying Machine Learning Classifiers](https://mlsanity.com/paper/2302.02667/) (2023)
- [ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement](https://mlsanity.com/paper/2303.10727/) (2023)
- [MetaMixer: A Regularization Strategy for Online Knowledge Distillation](https://mlsanity.com/paper/2303.07951/) (2023)
- [StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation](https://mlsanity.com/paper/2312.13223/) (2023)
- [TT-MPD: Test Time Model Pruning and Distillation](https://mlsanity.com/paper/2412.07114/) (2024)
- [Predicting the Reliability of an Image Classifier under Image Distortion](https://mlsanity.com/paper/2412.16881/) (2024)
- [PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models](https://mlsanity.com/paper/2506.00910/) (2025)
- [Diverse Image Priors for Black-box Data-free Knowledge Distillation](https://mlsanity.com/paper/2604.25794/) (2026)
- [Improving Diversity in Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2604.25795/) (2026)

## Abstract (arXiv)

We study how to train a student deep neural network for visual recognition by distilling knowledge from a blackbox teacher model in a data-efficient manner. Progress on this problem can significantly reduce the dependence on large-scale datasets for learning high-performing visual recognition models. There are two major challenges. One is that the number of queries into the teacher model should be minimized to save computational and/or financial costs. The other is that the number of images used for the knowledge distillation should be small; otherwise, it violates our expectation of reducing the dependence on large-scale datasets. To tackle these challenges, we propose an approach that blends mixup and active learning. The former effectively augments the few unlabeled images by a big pool of synthetic images sampled from the convex hull of the original images, and the latter actively chooses from the pool hard examples for the student neural network and query their labels from the teacher model. We validate our approach with extensive experiments.