# Quantifying the Value of Privileged Information Using a PAC-Bayesian Approach

arXiv: https://arxiv.org/abs/2609.12891 · HTML: https://mlsanity.com/paper/2609.12891/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Modality Distillation with Multiple Stream Networks for Action Recognition](https://mlsanity.com/paper/1806.07110/) (2018)
- [Deep Learning under Privileged Information Using Heteroscedastic Dropout](https://mlsanity.com/paper/1805.11614/) (2018)
- [Transfer and Marginalize: Explaining Away Label Noise with Privileged Information](https://mlsanity.com/paper/2202.09244/) (2022)
- [MIML-FCN+: Multi-instance Multi-label Learning via Fully Convolutional Networks with Privileged Information](https://mlsanity.com/paper/1702.08681/) (2017)
- [Mind the Nuisance: Gaussian Process Classification using Privileged Noise](https://mlsanity.com/paper/1407.0179/) (2014)
- [Pi-DUAL: Using Privileged Information to Distinguish Clean from Noisy Labels](https://mlsanity.com/paper/2310.06600/) (2023)

## Abstract (arXiv)

In practice, various learning scenarios provide access to auxiliary features exclusively during training. Incorporating such data to enhance model performance gave rise to a paradigm known as Learning Using Privileged Information (LUPI). While this extra information is intended to improve the resulting model, establishing a generalized, cohesive understanding of how privileged information (PI) transfers useful knowledge remains a challenge. Vapnik's original theory and subsequent works offer performance guarantees in certain cases, but these results are inherently per-algorithm and rely on setting-specific proof approaches. Consequently, a more general framework explaining how and when PI transfers useful knowledge is still missing. To bridge this gap, we introduce an algorithm-agnostic, information-theoretic approach based on the PAC-Bayes framework. Rather than asking whether a particular algorithm exploits PI, we ask how much value it could offer: comparing the tightest achievable risk bound with and without PI yields its potential - an upper limit on the extractable gain. We introduce a metric that quantifies this potential directly from empirical training risk, bypassing the need for test-time data access, and validate our findings in both supervised and unsupervised settings. The results demonstrate a robust correspondence between our training-time metric and true test-time performance gains. Ultimately, this work takes a necessary step toward an information-theoretic understanding of LUPI, and quantifying the potential of privileged features before committing to a model.