# Rethinking Knowledge Transfer in Learning Using Privileged Information

arXiv: https://arxiv.org/abs/2408.14319 · HTML: https://mlsanity.com/paper/2408.14319/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 12 of 12)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Learning with Privileged Information for Efficient Image Super-Resolution](https://mlsanity.com/paper/2007.07524/) (2020)
- [Learning with privileged information via adversarial discriminative modality distillation](https://mlsanity.com/paper/1810.08437/) (2018)
- [Deep Learning under Privileged Information Using Heteroscedastic Dropout](https://mlsanity.com/paper/1805.11614/) (2018)
- [Privileged Features Distillation at Taobao Recommendations](https://mlsanity.com/paper/1907.05171/) (2019)
- [Toward Understanding Privileged Features Distillation in Learning-to-Rank](https://mlsanity.com/paper/2209.08754/) (2022)
- [Transfer and Marginalize: Explaining Away Label Noise with Privileged Information](https://mlsanity.com/paper/2202.09244/) (2022)
- [Ad-load Balancing via Off-policy Learning in a Content Marketplace](https://mlsanity.com/paper/2309.11518/) (2023)
- [Patterns for Learning with Side Information](https://mlsanity.com/paper/1511.06429/) (2015)
- [When does Privileged Information Explain Away Label Noise?](https://mlsanity.com/paper/2303.01806/) (2023)

## Built on by (most cited first, 2 of 2)

- [Longitudinal Risk Prediction in Mammography with Privileged History Distillation](https://mlsanity.com/paper/2603.15814/) (2026)
- [TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration](https://mlsanity.com/paper/2608.10286/) (2026)

## Abstract (arXiv)

In supervised machine learning, privileged information (PI) is information that is unavailable at inference, but is accessible during training time. Research on learning using privileged information (LUPI) aims to transfer the knowledge captured in PI onto a model that can perform inference without PI. It seems that this extra bit of information ought to make the resulting model better. However, finding conclusive theoretical or empirical evidence that supports the ability to transfer knowledge using PI has been challenging. In this paper, we critically examine the assumptions underlying existing theoretical analyses and argue that there is little theoretical justification for when LUPI should work. We analyze LUPI methods and reveal that apparent improvements in empirical risk of existing research may not directly result from PI. Instead, these improvements often stem from dataset anomalies or modifications in model design misguidedly attributed to PI. Our experiments for a wide variety of application domains further demonstrate that state-of-the-art LUPI approaches fail to effectively transfer knowledge from PI. Thus, we advocate for practitioners to exercise caution when working with PI to avoid unintended inductive biases.