mlsanityData to 14 Oct 2026

Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More

arXiv 1904.10167 ↗First posted 23 Apr 2019cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate a novel deep-model reusing task. Our goal is to train a lightweight and versatile student model, without human-labelled annotations, that amalgamates the knowledge and masters the expertise of two pretrained teacher models working on heterogeneous problems, one on scene parsing and the other on depth estimation. To this end, we propose an innovative training strategy that learns the parameters of the student intertwined with the teachers, achieved by 'projecting' its amalgamated features onto each teacher's domain and computing the loss. We also introduce two options to generalize the proposed training strategy to handle three or more tasks simultaneously. The proposed scheme yields very encouraging results. As demonstrated on several benchmarks, the trained student model achieves results even superior to those of the teachers in their own expertise domains and on par with the state-of-the-art fully supervised models relying on human-labelled annotations.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 11; 2023: 3; 2024: 3; 2025: 1.

29 papers.

Sort:

By area:

  1. 2022Diffusion Probabilistic Model Made Slimcited by 121
  2. 2022Curriculum Temperature for Knowledge Distillationcited by 112
  3. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 86
  4. 2022Deep Model Reassemblycited by 78
  5. 2021Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularizationcited by 69
  6. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  7. 2022Factorizing Knowledge in Neural Networkscited by 59
  8. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  9. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Datacited by 32
  10. 2024Training-Free Pretrained Model Mergingcited by 32
  11. 2022Learning with Recoverable Forgettingcited by 30
  12. 2022Spot-adaptive Knowledge Distillationcited by 22
17 more
  1. 2021Heterogeneous Multi-task Learning with Expert Diversitycited by 18
  2. 2023BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-task Dense Predictionscited by 12
  3. 2021KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillationcited by 11
  4. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 7
  5. 2022Progressive Cross-modal Knowledge Distillation for Human Action Recognitioncited by 6
  6. 2022Knowledge Amalgamation for Object Detection with Transformerscited by 5
  7. 2021Edge-competing Pathological Liver Vessel Segmentation with Limited Labelscited by 3
  8. 2021Towards Communication-Efficient and Privacy-Preserving Federated Representation Learningcited by 3
  9. 2021Improved Knowledge Distillation via Adversarial Collaborationcited by 3
  10. 2025UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Blockcited by 3
  11. 2021Up to 100× Faster Data-free Knowledge Distillationcited by 2
  12. 2023Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removalcited by 2
  13. 2021Visual Boundary Knowledge Translation for Foreground Segmentationcited by 1
  14. 2022Optimizer Amalgamationcited by 1
  15. 2022Federated Selective Aggregation for Knowledge Amalgamationcited by 1
  16. 2022From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Modelscited by 1
  17. 2024Anno-incomplete Multi-dataset Detectioncited by 0