mlsanityData to 7 Oct 2026

Unifying distillation and privileged information

arXiv 1511.03643 ↗First posted 11 Nov 2015stat.ML cs.LGcited by 218

Not on the map. Why?

Abstract (arXiv)

Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretical and causal insight about the inner workings of generalized distillation, extend it to unsupervised, semisupervised and multitask learning scenarios, and illustrate its efficacy on a variety of numerical simulations on both synthetic and real-world data.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 43; 2022: 23; 2023: 35; 2024: 22; 2025: 20; 2026: 75.

218 papers.

Sort:

By area:

  1. 2021Towards Understanding Knowledge Distillationcited by 158
  2. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  3. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  4. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  5. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  6. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  7. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  8. 2021Annealing Knowledge Distillationcited by 61
  9. 2025Distillation Scaling Lawscited by 59
  10. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  11. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
  12. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
38 more
  1. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  2. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  3. 2021Teachers Do More Than Teach: Compressing Image-to-Image Modelscited by 37
  4. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  5. 2021BiMaL: Bijective Maximum Likelihood Approach to Domain Adaptation in Semantic Scene Segmentationcited by 31
  6. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  7. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  8. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  9. 2021Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillationcited by 28
  10. 2023Human Pose Estimation in Extremely Low-Light Conditionscited by 27
  11. 2023Adaptive Data-Free Quantizationcited by 26
  12. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  13. 2023Prototype Knowledge Distillation for Medical Segmentation with Missing Modalitycited by 24
  14. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  15. 2022Revisiting Self-Distillationcited by 23
  16. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  17. 2022Geometric Knowledge Distillation: Topology Compression for Graph Neural Networkscited by 22
  18. 2021Knowledge-Adaptation Priorscited by 20
  19. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 20
  20. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  21. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  22. 2025Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillationcited by 20
  23. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  24. 2021Motion-Augmented Self-Training for Video Recognition at Smaller Scalecited by 17
  25. 2021Churn Reduction via Distillationcited by 17
  26. 2021Few-Shot Learning with a Strong Teachercited by 16
  27. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  28. 2022Toward Understanding Privileged Features Distillation in Learning-to-Rankcited by 16
  29. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  30. 2022Transfer and Marginalize: Explaining Away Label Noise with Privileged Informationcited by 15
  31. 2021Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillationcited by 14
  32. 2024Understanding the Gains from Repeated Self-Distillationcited by 14
  33. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
  34. 2021Finding the Right Moment: Human-Assisted Trailer Creation via Task Compositioncited by 12
  35. 2022Knowledge Distillation: Bad Models Can Be Good Role Modelscited by 12
  36. 2022Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesiscited by 12
  37. 2023From the Lab to the Wild: Affect Modeling via Privileged Informationcited by 12
  38. 2025Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecificationcited by 12

50 most cited of 218 All 218 (JSON)