# Creating Training Sets via Weak Indirect Supervision

arXiv: https://arxiv.org/abs/2110.03484 · HTML: https://mlsanity.com/paper/2110.03484/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Training Complex Models with Multi-Task Weak Supervision](https://mlsanity.com/paper/1810.02840/) (2018)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale](https://mlsanity.com/paper/1812.00417/) (2018)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Learning From Noisy Singly-labeled Data](https://mlsanity.com/paper/1712.04577/) (2017)
- [Generating Multi-Agent Trajectories using Programmatic Weak Supervision](https://mlsanity.com/paper/1803.07612/) (2018)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs](https://mlsanity.com/paper/2007.02387/) (2020)
- [FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply](https://mlsanity.com/paper/2006.07512/) (2020)
- [Cross-Modal Data Programming Enables Rapid Medical Machine Learning](https://mlsanity.com/paper/1903.11101/) (2019)
- [AutoKnow: Self-Driving Knowledge Collection for Products of Thousands of Types](https://mlsanity.com/paper/2006.13473/) (2020)
- [NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extraction](https://mlsanity.com/paper/1909.02177/) (2019)
- [LSHTC: A Benchmark for Large-Scale Text Classification](https://mlsanity.com/paper/1503.08581/) (2015)
- [Ontology-driven weak supervision for clinical entity classification in electronic health records](https://mlsanity.com/paper/2008.01972/) (2020)
- [Multi-Resolution Weak Supervision for Sequential Data](https://mlsanity.com/paper/1910.09505/) (2019)
- [Learnability with Indirect Supervision Signals](https://mlsanity.com/paper/2006.08791/) (2020)
- [Estimation from Indirect Supervision with Linear Moments](https://mlsanity.com/paper/1608.03100/) (2016)
- [Finer Grained Entity Typing with TypeNet](https://mlsanity.com/paper/1711.05795/) (2017)
- [Learning from Indirect Observations](https://mlsanity.com/paper/1910.04394/) (2019)

## Built on by (most cited first, 14 of 14)

- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Losses over Labels: Weakly Supervised Learning via Direct Loss Construction](https://mlsanity.com/paper/2212.06921/) (2022)
- [Decoding the Narratives: Analyzing Personal Drug Experiences Shared on Reddit](https://mlsanity.com/paper/2406.12117/) (2024)
- [Understanding Programmatic Weak Supervision via Source-aware Influence Function](https://mlsanity.com/paper/2205.12879/) (2022)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification](https://mlsanity.com/paper/2210.03092/) (2022)
- [Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision](https://mlsanity.com/paper/2210.02724/) (2022)
- [Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs](https://mlsanity.com/paper/2409.10702/) (2024)
- [Binary Classification with Positive Labeling Sources](https://mlsanity.com/paper/2208.01704/) (2022)
- [Leveraging Large Language Models for Structure Learning in Prompted Weak Supervision](https://mlsanity.com/paper/2402.01867/) (2024)
- [WeShap: Weak Supervision Source Evaluation with Shapley Values](https://mlsanity.com/paper/2406.11010/) (2024)

## Abstract (arXiv)

Creating labeled training sets has become one of the major roadblocks in machine learning. To address this, recent Weak Supervision (WS) frameworks synthesize training labels from multiple potentially noisy supervision sources. However, existing frameworks are restricted to supervision sources that share the same output space as the target task. To extend the scope of usable sources, we formulate Weak Indirect Supervision (WIS), a new research problem for automatically synthesizing training labels based on indirect supervision sources that have different output label spaces. To overcome the challenge of mismatched output spaces, we develop a probabilistic modeling approach, PLRM, which uses user-provided label relations to model and leverage indirect supervision sources. Moreover, we provide a theoretically-principled test of the distinguishability of PLRM for unseen labels, along with a generalization bound. On both image and text classification tasks as well as an industrial advertising application, we demonstrate the advantages of PLRM by outperforming baselines by a margin of 2%-9%.