mlsanityData to 7 Oct 2026

A Survey on Programmatic Weak Supervision

arXiv 2202.05433 ↗First posted 11 Feb 2022cs.LG cs.AI stat.APcited by 66

Not on the map. Why?

Abstract (arXiv)

Labeling training data has become one of the major roadblocks to using machine learning. Among various weak supervision paradigms, programmatic weak supervision (PWS) has achieved remarkable success in easing the manual labeling bottleneck by programmatically synthesizing training labels from multiple potentially noisy supervision sources. This paper presents a comprehensive survey of recent advances in PWS. In particular, we give a brief introduction of the PWS learning paradigm, and review representative approaches for each component within PWS's learning workflow. In addition, we discuss complementary learning paradigms for tackling limited labeled data scenarios and how these related approaches can be used in conjunction with PWS. Finally, we identify several critical challenges that remain under-explored in the area to hopefully inspire future research directions in the field.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 44 of its 72 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

44 of 72 references.

References · 44

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  3. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  4. 2020A Survey of Deep Active Learningcited by 547
  5. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  6. 2018A Survey of Unsupervised Deep Domain Adaptationcited by 361
  7. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  8. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  9. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  10. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  11. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  12. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
32 more
  1. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  2. 2020BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervisioncited by 91
  3. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  4. 2018Weakly-Supervised Neural Text Classificationcited by 66
  5. 2020Learning from Rules Generalizing Labeled Exemplarscited by 59
  6. 2018Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scalecited by 52
  7. 2020Interactive Weak Supervision: Learning Useful Heuristics for Data Labelingcited by 52
  8. 2017Learning the Structure of Generative Models without Labeled Datacited by 50
  9. 2017Learning From Noisy Singly-labeled Datacited by 50
  10. 2021Self-Training with Weak Supervisioncited by 49
  11. 2020Named Entity Recognition without Labelled Data: A Weak Supervision Approachcited by 46
  12. 2020Denoising Multi-Source Weak Supervision for Neural Text Classificationcited by 42
  13. 2021End-to-End Weak Supervisioncited by 33
  14. 2019Learning Dependency Structures for Weak Supervision Modelscited by 32
  15. 2019Scene Graph Prediction with Limited Labelscited by 32
  16. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  17. 2021Universalizing Weak Supervisioncited by 27
  18. 2019Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labelingcited by 22
  19. 2021Weakly Supervised Named Entity Tagging with Learnable Logical Rulescited by 21
  20. 2017Inferring Generative Model Structure with Static Analysiscited by 18
  21. 2021DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samplescited by 18
  22. 2019Multi-Resolution Weak Supervision for Sequential Datacited by 17
  23. 2020Semi-Supervised Data Programming with Subset Selectioncited by 16
  24. 2020Adaptive Rule Discovery for Labeling Text Datacited by 15
  25. 2021BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognitioncited by 15
  26. 2021Learning from Multiple Noisy Partial Labelerscited by 14
  27. 2021Creating Training Sets via Weak Indirect Supervisioncited by 14
  28. 2021GLaRA: Graph-based Labeling Rule Augmentation for Weakly Supervised Named Entity Recognitioncited by 12
  29. 2021Active WeaSuL: Improving Weak Supervision with Active Learningcited by 12
  30. 2018Adversarial Label Learningcited by 9
  31. 2021Dependency Structure Misspecification in Multi-Source Weak Supervision Modelscited by 6
  32. 2020Iterative Data Programming for Expanding Text Classification Corporacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 23; 2024: 15; 2025: 7; 2026: 4.

66 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  2. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  3. 2024Are We Done with MMLU?cited by 178
  4. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  5. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  6. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  7. 2022Unsupervised Model Selection for Time-series Anomaly Detectioncited by 24
  8. 2022Learning Hyper Label Model for Programmatic Weak Supervisioncited by 16
  9. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  10. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  11. 2022PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learningcited by 14
  12. 2022Losses over Labels: Weakly Supervised Learning via Direct Loss Constructioncited by 11
38 more
  1. 2022Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programmingcited by 10
  2. 2023Unsupervised classification to improve the quality of a bird song recording datasetcited by 10
  3. 2023Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurationscited by 9
  4. 2023Improved Active Multi-Task Representation Learning via Lassocited by 9
  5. 2022Understanding Programmatic Weak Supervision via Source-aware Influence Functioncited by 8
  6. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  7. 2023Alfred: A System for Prompted Weak Supervisioncited by 8
  8. 2023Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweetscited by 8
  9. 2022Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classificationcited by 7
  10. 2023VideoPro: A Visual Analytics Approach for Interactive Video Programmingcited by 7
  11. 2024Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processingcited by 7
  12. 2022Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervisioncited by 6
  13. 2023Can Large Language Models Design Accurate Label Functions?cited by 6
  14. 2022Satellite Monitoring of Terrestrial Plastic Wastecited by 4
  15. 2023Weak Supervision for Label Efficient Visual Bug Detectioncited by 4
  16. 2023Weak Supervision Performance Evaluation via Partial Identificationcited by 4
  17. 2024LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systemscited by 4
  18. 2024Data Collection and Labeling Techniques for Machine Learningcited by 4
  19. 2023Mitigating Source Bias for Fairer Weak Supervisioncited by 3
  20. 2023Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labelercited by 3
  21. 2024WISER: Weak supervISion and supErvised Representation learning to improve drug response prediction in cancercited by 3
  22. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3
  23. 2022Binary Classification with Positive Labeling Sourcescited by 2
  24. 2024Employing Label Models on ChatGPT Answers Improves Legal Text Entailment Performancecited by 2
  25. 2024LLM-assisted Labeling Function Generation for Semantic Type Detectioncited by 2
  26. 2024Interactive Machine Teaching by Labeling Rules and Instancescited by 2
  27. 2022XPASC: Measuring Generalization in Weak Supervision by Explainability and Associationcited by 1
  28. 2023Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documentscited by 1
  29. 2023An Adaptive Method for Weak Supervision with Drifting Datacited by 1
  30. 2023Fusing Conditional Submodular GAN and Programmatic Weak Supervisioncited by 1
  31. 2024Leveraging Large Language Models for Structure Learning in Prompted Weak Supervisioncited by 1
  32. 2024Enhancing Multi-Domain Automatic Short Answer Grading through an Explainable Neuro-Symbolic Pipelinecited by 1
  33. 2024WeShap: Weak Supervision Source Evaluation with Shapley Valuescited by 1
  34. 2025Refining Labeling Functions with Limited Labeled Datacited by 1
  35. 2025SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Datacited by 1
  36. 2025Reliable Programmatic Weak Supervision with Confidence Intervals for Label Probabilitiescited by 1
  37. 2025Learning from Interval Targetscited by 1
  38. 2022Generative Modeling Helps Weak Supervision (and Vice Versa)cited by 0

50 most cited of 66 All 66 (JSON)