# Foreseeing the Benefits of Incidental Supervision

arXiv: https://arxiv.org/abs/2006.05500 · HTML: https://mlsanity.com/paper/2006.05500/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 7 of 7)

- [Summarization as Indirect Supervision for Relation Extraction](https://mlsanity.com/paper/2205.09837/) (2022)
- [Weighted Training for Cross-Task Learning](https://mlsanity.com/paper/2105.14095/) (2021)
- [Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?](https://mlsanity.com/paper/2212.10784/) (2022)
- [To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering](https://mlsanity.com/paper/2212.10381/) (2022)
- [PInKS: Preconditioned Commonsense Inference with Minimal Supervision](https://mlsanity.com/paper/2206.07920/) (2022)
- [Dense Retrieval as Indirect Supervision for Large-space Decision Making](https://mlsanity.com/paper/2310.18619/) (2023)
- [Surveying the Dead Minds: Historical-Psychological Text Analysis with Contextualized Construct Representation (CCR) for Classical Chinese](https://mlsanity.com/paper/2403.00509/) (2024)

## Abstract (arXiv)

Real-world applications often require improved models by leveraging a range of cheap incidental supervision signals. These could include partial labels, noisy labels, knowledge-based constraints, and cross-domain or cross-task annotations -- all having statistical associations with gold annotations but not exactly the same. However, we currently lack a principled way to measure the benefits of these signals to a given target task, and the common practice of evaluating these benefits is through exhaustive experiments with various models and hyperparameters. This paper studies whether we can, in a single framework, quantify the benefits of various types of incidental signals for a given target task without going through combinatorial experiments. We propose a unified PAC-Bayesian motivated informativeness measure, PABI, that characterizes the uncertainty reduction provided by incidental supervision signals. We demonstrate PABI's effectiveness by quantifying the value added by various types of incidental signals to sequence tagging tasks. Experiments on named entity recognition (NER) and question answering (QA) show that PABI's predictions correlate well with learning performance, providing a promising way to determine, ahead of learning, which supervision signals would be beneficial.