mlsanityData to 14 Oct 2026

WRENCH: A Comprehensive Benchmark for Weak Supervision

arXiv 2109.11377 ↗First posted 23 Sep 2021cs.LG cs.AI cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

Recent Weak Supervision (WS) approaches have had widespread success in easing the bottleneck of labeling training data for machine learning by synthesizing labels from multiple potentially noisy supervision sources. However, proper measurement and analysis of these approaches remain a challenge. First, datasets used in existing works are often private and/or custom, limiting standardization. Second, WS datasets with the same name and base data often vary in terms of the labels and weak supervision sources used, a significant "hidden" source of evaluation variance. Finally, WS studies often diverge in terms of the evaluation protocol and ablations used. To address these problems, we introduce a benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches. It consists of 22 varied real-world datasets for classification and sequence tagging; a range of real, synthetic, and procedurally-generated weak supervision sources; and a modular, extensible framework for WS evaluation, including implementations for popular WS methods. We use WRENCH to conduct extensive comparisons over more than 120 method variants to demonstrate its efficacy as a benchmark platform. The code is available at https://github.com/JieyuZ2/wrench.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 55 of its 114 references that are in the map. First posted on arXiv 23 Sep 2021.

Builds on

55 of 114 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  6. 2019When Does Label Smoothing Help?cited by 988
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
  9. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  10. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  11. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  12. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
38 more
  1. 2020BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervisioncited by 91
  2. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  3. 2019Enriching Pre-trained Language Model with Entity Information for Relation Classificationcited by 82
  4. 2020MoPro: Webly Supervised Learning with Momentum Prototypescited by 66
  5. 2018Learning Named Entity Tagger using Domain-Specific Dictionarycited by 65
  6. 2019SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominalscited by 63
  7. 2020Learning from Rules Generalizing Labeled Exemplarscited by 59
  8. 2020Interactive Weak Supervision: Learning Useful Heuristics for Data Labelingcited by 52
  9. 2017Learning the Structure of Generative Models without Labeled Datacited by 50
  10. 2021Self-Training with Weak Supervisioncited by 49
  11. 2018Generating Multi-Agent Trajectories using Programmatic Weak Supervisioncited by 46
  12. 2019Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learningcited by 46
  13. 2020Named Entity Recognition without Labelled Data: A Weak Supervision Approachcited by 46
  14. 2020Denoising Multi-Source Weak Supervision for Neural Text Classificationcited by 42
  15. 2019Learning Dependency Structures for Weak Supervision Modelscited by 32
  16. 2019Cross-Modal Data Programming Enables Rapid Medical Machine Learningcited by 32
  17. 2019Scene Graph Prediction with Limited Labelscited by 32
  18. 2017SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Datacited by 28
  19. 2021Noisy-Labeled NER with Confidence Estimationcited by 28
  20. 2017PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCNcited by 25
  21. 2021Visual Distant Supervision for Scene Graph Generationcited by 25
  22. 2019NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extractioncited by 24
  23. 2020Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake Newscited by 24
  24. 2019Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labelingcited by 22
  25. 2021Weakly Supervised Named Entity Tagging with Learnable Logical Rulescited by 21
  26. 2019GOGGLES: Automatic Image Labeling with Affinity Codingcited by 19
  27. 2017Inferring Generative Model Structure with Static Analysiscited by 18
  28. 2020Ontology-driven weak supervision for clinical entity classification in electronic health recordscited by 18
  29. 2019Multi-Resolution Weak Supervision for Sequential Datacited by 17
  30. 2020Learning with Weak Supervision for Email Intent Detectioncited by 17
  31. 2021skweak: Weak Supervision Made Easy for NLPcited by 17
  32. 2020Semi-Supervised Data Programming with Subset Selectioncited by 16
  33. 2020Adaptive Rule Discovery for Labeling Text Datacited by 15
  34. 2021BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognitioncited by 15
  35. 2017Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrixcited by 14
  36. 2019Rekall: Specifying Video Events using Compositions of Spatiotemporal Labelscited by 14
  37. 2021Analysing the Noise Model Error for Realistic Noisy Label Datacited by 14
  38. 2021QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extractioncited by 13

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 30; 2023: 33; 2024: 15; 2025: 9; 2026: 7.

96 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  2. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  3. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  4. 2022A Survey on Programmatic Weak Supervisioncited by 66
  5. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 52
  6. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  7. 2022To Aggregate or Not? Learning with Separate Noisy Labelscited by 29
  8. 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27
  9. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  10. 2023Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidancecited by 24
  11. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  12. 2023Identification of Negative Transfers in Multitask Learning Using Surrogate Modelscited by 20
38 more
  1. 2023LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learningcited by 20
  2. 2023Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Modelscited by 20
  3. 2022Training Subset Selection for Weak Supervisioncited by 19
  4. 2022Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervisioncited by 18
  5. 2023Making Scalable Meta Learning Practicalcited by 17
  6. 2022Learning Hyper Label Model for Programmatic Weak Supervisioncited by 16
  7. 2024A Simple but Effective Approach to Improve Structured Language Model Output for Information Extractioncited by 16
  8. 2021Creating Training Sets via Weak Indirect Supervisioncited by 14
  9. 2022PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learningcited by 14
  10. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  11. 2022Losses over Labels: Weakly Supervised Learning via Direct Loss Constructioncited by 11
  12. 2025Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistancecited by 11
  13. 2022Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programmingcited by 10
  14. 2023DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modelingcited by 10
  15. 2023TART: A plug-and-play Transformer module for task-agnostic reasoningcited by 10
  16. 2024The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotatorscited by 10
  17. 2022Label Propagation with Weak Supervisioncited by 9
  18. 2022Understanding Programmatic Weak Supervision via Source-aware Influence Functioncited by 8
  19. 2023Learning with Explanation Constraintscited by 8
  20. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  21. 2023Alfred: A System for Prompted Weak Supervisioncited by 8
  22. 2023Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Trainingcited by 8
  23. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  24. 2022Meta Self-Refinement for Robust Learning with Weak Supervisioncited by 7
  25. 2022AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labelscited by 7
  26. 2022Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classificationcited by 7
  27. 2022Automatic Rule Induction for Interpretable Semi-Supervised Learningcited by 6
  28. 2022Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervisioncited by 6
  29. 2023A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approachescited by 6
  30. 2023Can Large Language Models Design Accurate Label Functions?cited by 6
  31. 2022Ground Truth Inference for Weakly Supervised Entity Matchingcited by 5
  32. 2023Pareto Optimal Learning for Estimating Large Language Model Errorscited by 5
  33. 2022Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognitioncited by 4
  34. 2022Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Predictioncited by 4
  35. 2022SepLL: Separating Latent Class Labels from Weak Supervision Noisecited by 4
  36. 2023Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removalcited by 4
  37. 2023Weak Supervision Performance Evaluation via Partial Identificationcited by 4
  38. 2024Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysiscited by 4

50 most cited of 96 All 96 (JSON)