# Reliable Programmatic Weak Supervision with Confidence Intervals for Label Probabilities

arXiv: https://arxiv.org/abs/2508.03896 · HTML: https://mlsanity.com/paper/2508.03896/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 27 of 27)

- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Distribution-free binary classification: prediction sets, confidence intervals and calibration](https://mlsanity.com/paper/2006.10564/) (2020)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Learning from Rules Generalizing Labeled Exemplars](https://mlsanity.com/paper/2004.06025/) (2020)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Denoising Multi-Source Weak Supervision for Neural Text Classification](https://mlsanity.com/paper/2010.04582/) (2020)
- [Learning Dependency Structures for Weak Supervision Models](https://mlsanity.com/paper/1903.05844/) (2019)
- [Universalizing Weak Supervision](https://mlsanity.com/paper/2112.03865/) (2021)
- [Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing](https://mlsanity.com/paper/1411.4086/) (2014)
- [GOGGLES: Automatic Image Labeling with Affinity Coding](https://mlsanity.com/paper/1903.04552/) (2019)
- [Incremental Knowledge Base Construction Using DeepDive](https://mlsanity.com/paper/1502.00731/) (2015)
- [Wasserstein Adversarial Regularization (WAR) on label noise](https://mlsanity.com/paper/1904.03936/) (2019)
- [Learning Hyper Label Model for Programmatic Weak Supervision](https://mlsanity.com/paper/2207.13545/) (2022)
- [Generalized Maximum Entropy for Supervised Classification](https://mlsanity.com/paper/2007.05447/) (2020)
- [Minimax Classification with 0-1 Loss and Performance Guarantees](https://mlsanity.com/paper/2010.07964/) (2020)
- [Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds](https://mlsanity.com/paper/2306.03116/) (2023)
- [Lifting Weak Supervision To Structured Prediction](https://mlsanity.com/paper/2211.13375/) (2022)
- [Adversarial Label Learning](https://mlsanity.com/paper/1805.08877/) (2018)
- [Optimal Binary Classifier Aggregation for General Losses](https://mlsanity.com/paper/1510.00452/) (2015)
- [Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision](https://mlsanity.com/paper/2210.02724/) (2022)
- [Minimax Forward and Backward Learning of Evolving Tasks with Performance Guarantees](https://mlsanity.com/paper/2310.15974/) (2023)
- [Convergence Behavior of an Adversarial Weak Supervision Method](https://mlsanity.com/paper/2405.16013/) (2024)

## Built on by (most cited first, 1 of 1)

- [Structured Exploration and Exploitation of Label Functions for Automated Data Annotation](https://mlsanity.com/paper/2604.08578/) (2026)

## Abstract (arXiv)

The accurate labeling of datasets is often both costly and time-consuming. Given an unlabeled dataset, programmatic weak supervision obtains probabilistic predictions for the labels by leveraging multiple weak labeling functions (LFs) that provide rough guesses for labels. Weak LFs commonly provide guesses with assorted types and unknown interdependences that can result in unreliable predictions. Furthermore, existing techniques for programmatic weak supervision cannot provide assessments for the reliability of the probabilistic predictions for labels. This paper presents a methodology for programmatic weak supervision that can provide confidence intervals for label probabilities and obtain more reliable predictions. In particular, the methods proposed use uncertainty sets of distributions that encapsulate the information provided by LFs with unrestricted behavior and typology. Experiments on multiple benchmark datasets show the improvement of the presented methods over the state-of-the-art and the practicality of the confidence intervals presented.