# Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

arXiv: https://arxiv.org/abs/2003.09088 · HTML: https://mlsanity.com/paper/2003.09088/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2020.

## Built on by (most cited first, 30 of 32)

- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers](https://mlsanity.com/paper/2204.12997/) (2022)
- [Zero-shot Adversarial Quantization](https://mlsanity.com/paper/2103.15263/) (2021)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Learning with Recoverable Forgetting](https://mlsanity.com/paper/2207.08224/) (2022)
- [TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models](https://mlsanity.com/paper/2301.01296/) (2023)
- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [A survey on GANs for computer vision: Recent research, analysis and taxonomy](https://mlsanity.com/paper/2203.11242/) (2022)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition](https://mlsanity.com/paper/2305.17421/) (2023)
- [Revisiting Data-Free Knowledge Distillation with Poisoned Teachers](https://mlsanity.com/paper/2306.02368/) (2023)
- [Sampling to Distill: Knowledge Transfer from Open-World Data](https://mlsanity.com/paper/2307.16601/) (2023)
- [The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image](https://mlsanity.com/paper/2112.00725/) (2021)
- [Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation](https://mlsanity.com/paper/2303.11611/) (2023)
- [Training Generative Adversarial Networks in One Stage](https://mlsanity.com/paper/2103.00430/) (2021)
- [Partial Network Cloning](https://mlsanity.com/paper/2303.10597/) (2023)
- [Representation Consolidation for Training Expert Students](https://mlsanity.com/paper/2107.08039/) (2021)
- [Knowledge Amalgamation for Object Detection with Transformers](https://mlsanity.com/paper/2203.03187/) (2022)
- [Synthetic data generation method for data-free knowledge distillation in regression neural networks](https://mlsanity.com/paper/2301.04338/) (2023)
- [Aligning Logits Generatively for Principled Black-Box Knowledge Distillation](https://mlsanity.com/paper/2205.10490/) (2022)
- [Encapsulating Knowledge in One Prompt](https://mlsanity.com/paper/2407.11902/) (2024)
- [Optimizer Amalgamation](https://mlsanity.com/paper/2203.06474/) (2022)
- [Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentation](https://mlsanity.com/paper/2412.08116/) (2024)
- [Distilling A Universal Expert from Clustered Federated Learning](https://mlsanity.com/paper/2506.20285/) (2025)
- [Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning](https://mlsanity.com/paper/2205.11071/) (2022)
- [Tackling Intertwined Data and Device Heterogeneities in Federated Learning with Unlimited Staleness](https://mlsanity.com/paper/2309.13536/) (2023)

## Abstract (arXiv)

Recent advances in deep learning have provided procedures for learning one network to amalgamate multiple streams of knowledge from the pre-trained Convolutional Neural Network (CNN) models, thus reduce the annotation cost. However, almost all existing methods demand massive training data, which may be unavailable due to privacy or transmission issues. In this paper, we propose a data-free knowledge amalgamate strategy to craft a well-behaved multi-task student network from multiple single/multi-task teachers. The main idea is to construct the group-stack generative adversarial networks (GANs) which have two dual generators. First one generator is trained to collect the knowledge by reconstructing the images approximating the original dataset utilized for pre-training the teachers. Then a dual generator is trained by taking the output from the former generator as input. Finally we treat the dual part generator as the target network and regroup it. As demonstrated on several benchmarks of multi-label classification, the proposed method without any training data achieves the surprisingly competitive results, even compared with some full-supervised methods.