# An Adaptive Method for Weak Supervision with Drifting Data

arXiv: https://arxiv.org/abs/2306.01658 · HTML: https://mlsanity.com/paper/2306.01658/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 73 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Training Complex Models with Multi-Task Weak Supervision](https://mlsanity.com/paper/1810.02840/) (2018)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach](https://mlsanity.com/paper/2102.05406/) (2021)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free](https://mlsanity.com/paper/1902.00980/) (2019)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale](https://mlsanity.com/paper/1812.00417/) (2018)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation](https://mlsanity.com/paper/2201.04038/) (2022)
- [Learning Dependency Structures for Weak Supervision Models](https://mlsanity.com/paper/1903.05844/) (2019)
- [Scene Graph Prediction with Limited Labels](https://mlsanity.com/paper/1904.11622/) (2019)
- [Universalizing Weak Supervision](https://mlsanity.com/paper/2112.03865/) (2021)
- [Adapting to Online Label Shift with Provable Guarantees](https://mlsanity.com/paper/2207.02121/) (2022)
- [Ranking and combining multiple predictors without labeled data](https://mlsanity.com/paper/1303.3257/) (2013)
- [Inferring Generative Model Structure with Static Analysis](https://mlsanity.com/paper/1709.02477/) (2017)
- [Ontology-driven weak supervision for clinical entity classification in electronic health records](https://mlsanity.com/paper/2008.01972/) (2020)
- [Multi-Resolution Weak Supervision for Sequential Data](https://mlsanity.com/paper/1910.09505/) (2019)
- [An Adaptive Algorithm for Learning with Unknown Distribution Drift](https://mlsanity.com/paper/2305.02252/) (2023)
- [Statistical Learning under Nonstationary Mixing Processes](https://mlsanity.com/paper/1512.08064/) (2015)
- [Optimally Combining Classifiers Using Unlabeled Data](https://mlsanity.com/paper/1503.01811/) (2015)
- [Adversarial Label Learning](https://mlsanity.com/paper/1805.08877/) (2018)

## Built on by (most cited first, 1 of 1)

- [Model Assessment and Selection under Temporal Distribution Shift](https://mlsanity.com/paper/2402.08672/) (2024)

## Abstract (arXiv)

We introduce an adaptive method with formal quality guarantees for weak supervision in a non-stationary setting. Our goal is to infer the unknown labels of a sequence of data by using weak supervision sources that provide independent noisy signals of the correct classification for each data point. This setting includes crowdsourcing and programmatic weak supervision. We focus on the non-stationary case, where the accuracy of the weak supervision sources can drift over time, e.g., because of changes in the underlying data distribution. Due to the drift, older data could provide misleading information to infer the label of the current data point. Previous work relied on a priori assumptions on the magnitude of the drift to decide how much data to use from the past. In contrast, our algorithm does not require any assumptions on the drift, and it adapts based on the input by dynamically varying its window size. In particular, at each step, our algorithm estimates the current accuracies of the weak supervision sources by identifying a window of past observations that guarantees a near-optimal minimization of the trade-off between the error due to the variance of the estimation and the error due to the drift. Experiments on synthetic and real-world labelers show that our approach adapts to the drift.