# Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling

arXiv: https://arxiv.org/abs/2012.06046 · HTML: https://mlsanity.com/paper/2012.06046/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 30 of 52)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Explore, Establish, Exploit: Red Teaming Language Models from Scratch](https://mlsanity.com/paper/2306.09442/) (2023)
- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information](https://mlsanity.com/paper/2104.09460/) (2021)
- [End-to-End Weak Supervision](https://mlsanity.com/paper/2107.02233/) (2021)
- [DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples](https://mlsanity.com/paper/2110.13740/) (2021)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning](https://mlsanity.com/paper/2203.09735/) (2022)
- [Active WeaSuL: Improving Weak Supervision with Active Learning](https://mlsanity.com/paper/2104.14847/) (2021)
- [Generalizing Bayesian Optimization with Decision-theoretic Entropies](https://mlsanity.com/paper/2210.01383/) (2022)
- [Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming](https://mlsanity.com/paper/2203.01382/) (2022)
- [The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators](https://mlsanity.com/paper/2407.11004/) (2024)
- [Weak Supervision for Affordable Modeling of Electrocardiogram Data](https://mlsanity.com/paper/2201.02936/) (2022)
- [Understanding Programmatic Weak Supervision via Source-aware Influence Function](https://mlsanity.com/paper/2205.12879/) (2022)
- [AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels](https://mlsanity.com/paper/2208.14362/) (2022)
- [Dependency Structure Misspecification in Multi-Source Weak Supervision Models](https://mlsanity.com/paper/2106.10302/) (2021)
- [Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis](https://mlsanity.com/paper/2111.15186/) (2021)
- [Automatic Rule Induction for Interpretable Semi-Supervised Learning](https://mlsanity.com/paper/2205.09067/) (2022)
- [Can Large Language Models Design Accurate Label Functions?](https://mlsanity.com/paper/2311.00739/) (2023)
- [TagRuler: Interactive Tool for Span-Level Data Programming by Demonstration](https://mlsanity.com/paper/2106.12767/) (2021)
- [Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition](https://mlsanity.com/paper/2205.14228/) (2022)
- [Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction](https://mlsanity.com/paper/2206.13749/) (2022)
- [Making Binary Classification from Multiple Unlabeled Datasets Almost Free of Supervision](https://mlsanity.com/paper/2306.07036/) (2023)
- [Weak Supervision for Label Efficient Visual Bug Detection](https://mlsanity.com/paper/2309.11077/) (2023)
- [ScriptoriumWS: A Code Generation Assistant for Weak Supervision](https://mlsanity.com/paper/2502.12366/) (2025)
- [CHEF: A Cheap and Fast Pipeline for Iteratively Cleaning Label Uncertainties (Technical Report)](https://mlsanity.com/paper/2107.08588/) (2021)

## Abstract (arXiv)

Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics. This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce. One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and domain expertise from those who hand-craft them, a process which can be tedious and subjective. We develop the first framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic. Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels. We conduct user studies, which show that users are able to effectively provide feedback on heuristics and that test set results track the performance of simulated oracles.