# Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More

arXiv: https://arxiv.org/abs/1904.10167 · HTML: https://mlsanity.com/paper/1904.10167/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2019.

## Built on by (most cited first, 29 of 29)

- [Diffusion Probabilistic Model Made Slim](https://mlsanity.com/paper/2211.17106/) (2022)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.13540/) (2023)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization](https://mlsanity.com/paper/2103.15537/) (2021)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Factorizing Knowledge in Neural Networks](https://mlsanity.com/paper/2207.03337/) (2022)
- [Online Knowledge Distillation for Efficient Pose Estimation](https://mlsanity.com/paper/2108.02092/) (2021)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Learning with Recoverable Forgetting](https://mlsanity.com/paper/2207.08224/) (2022)
- [Spot-adaptive Knowledge Distillation](https://mlsanity.com/paper/2205.02399/) (2022)
- [Heterogeneous Multi-task Learning with Expert Diversity](https://mlsanity.com/paper/2106.10595/) (2021)
- [BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-task Dense Predictions](https://mlsanity.com/paper/2312.13514/) (2023)
- [KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation](https://mlsanity.com/paper/2105.04181/) (2021)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Progressive Cross-modal Knowledge Distillation for Human Action Recognition](https://mlsanity.com/paper/2208.08090/) (2022)
- [Knowledge Amalgamation for Object Detection with Transformers](https://mlsanity.com/paper/2203.03187/) (2022)
- [Edge-competing Pathological Liver Vessel Segmentation with Limited Labels](https://mlsanity.com/paper/2108.00384/) (2021)
- [Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning](https://mlsanity.com/paper/2109.14611/) (2021)
- [Improved Knowledge Distillation via Adversarial Collaboration](https://mlsanity.com/paper/2111.14356/) (2021)
- [UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Block](https://mlsanity.com/paper/2507.19948/) (2025)
- [Up to 100× Faster Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.06253/) (2021)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [Visual Boundary Knowledge Translation for Foreground Segmentation](https://mlsanity.com/paper/2108.00379/) (2021)
- [Optimizer Amalgamation](https://mlsanity.com/paper/2203.06474/) (2022)
- [Federated Selective Aggregation for Knowledge Amalgamation](https://mlsanity.com/paper/2207.13309/) (2022)
- [From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models](https://mlsanity.com/paper/2210.05230/) (2022)
- [Anno-incomplete Multi-dataset Detection](https://mlsanity.com/paper/2408.16247/) (2024)

## Abstract (arXiv)

In this paper, we investigate a novel deep-model reusing task. Our goal is to train a lightweight and versatile student model, without human-labelled annotations, that amalgamates the knowledge and masters the expertise of two pretrained teacher models working on heterogeneous problems, one on scene parsing and the other on depth estimation. To this end, we propose an innovative training strategy that learns the parameters of the student intertwined with the teachers, achieved by 'projecting' its amalgamated features onto each teacher's domain and computing the loss. We also introduce two options to generalize the proposed training strategy to handle three or more tasks simultaneously. The proposed scheme yields very encouraging results. As demonstrated on several benchmarks, the trained student model achieves results even superior to those of the teachers in their own expertise domains and on par with the state-of-the-art fully supervised models relying on human-labelled annotations.