mlsanityData to 14 Oct 2026

Language Models in the Loop: Incorporating Prompting into Weak Supervision

arXiv 2205.02318 ↗First posted 4 May 2022cs.LG cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Rather than apply the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework. To create a classifier, we first prompt the model to answer multiple distinct queries about an example and define how the possible responses should be mapped to votes for labels and abstentions. We then denoise these noisy label sources using the Snorkel system and train an end classifier with the resulting training data. Our experimental evaluation shows that prompting large language models within a weak supervision framework can provide significant gains in accuracy. On the WRENCH weak supervision benchmark, this approach can significantly improve over zero-shot performance, an average 19.5% reduction in errors. We also find that this approach produces classifiers with comparable or superior accuracy to those trained from hand-engineered rules.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 53 of its 79 references that are in the map. First posted on arXiv 4 May 2022.

Builds on

53 of 79 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  10. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  11. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  12. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  13. 2021Generating Datasets with Pretrained Language Modelscited by 193
  14. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  15. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  16. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 96
  17. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  18. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 67
  19. 2022A Survey on Programmatic Weak Supervisioncited by 66
  20. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 62
  21. 2022RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extractioncited by 60
  22. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 58
  23. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  24. 2018Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scalecited by 52
  25. 2017Learning the Structure of Generative Models without Labeled Datacited by 50
  26. 2021Self-Training with Weak Supervisioncited by 49
  27. 2019Learning Dependency Structures for Weak Supervision Modelscited by 32
  28. 2019Cross-Modal Data Programming Enables Rapid Medical Machine Learningcited by 32
  29. 2021Universalizing Weak Supervisioncited by 27
  30. 2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24
  31. 2020Ontology-driven weak supervision for clinical entity classification in electronic health recordscited by 18
  32. 2022Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervisioncited by 18
  33. 2019Multi-Resolution Weak Supervision for Sequential Datacited by 17
  34. 2021Learning from Multiple Noisy Partial Labelerscited by 14
  35. 2021Creating Training Sets via Weak Indirect Supervisioncited by 14
  36. 2021Active WeaSuL: Improving Weak Supervision with Active Learningcited by 12
  37. 2020Constrained Labeling for Weakly Supervised Learningcited by 10
  38. 2020Active and Incremental Learning with Weak Supervisioncited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 20; 2024: 16; 2025: 4; 2026: 6.

52 papers.

Sort:

By area:

  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  3. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  4. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  5. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 111
  6. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  7. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
  8. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 72
  9. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  10. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  11. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  12. 2023Mixed Distillation Helps Smaller Language Model Better Reasoningcited by 19
38 more
  1. 2024Small Language Models are Good Too: An Empirical Study of Zero-Shot Classificationcited by 18
  2. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  3. 2023Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signalscited by 14
  4. 2023FreeAL: Towards Human-Free Active Learning in the Era of Large Language Modelscited by 14
  5. 2023Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Eventscited by 11
  6. 2024Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decodingcited by 11
  7. 2024LLM Chain Ensembles for Scalable and Accurate Data Annotationcited by 11
  8. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  9. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  10. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  11. 2023Alfred: A System for Prompted Weak Supervisioncited by 8
  12. 2024Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processingcited by 8
  13. 2024MAO: A Framework for Process Model Generation with Multi-Agent Orchestrationcited by 8
  14. 2022AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labelscited by 7
  15. 2023Towards an On-device Agent for Text Rewritingcited by 7
  16. 2023PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Trainingcited by 6
  17. 2023Weak Supervision Performance Evaluation via Partial Identificationcited by 4
  18. 2023Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learningcited by 3
  19. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Executioncited by 3
  20. 2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
  21. 2025ScriptoriumWS: A Code Generation Assistant for Weak Supervisioncited by 3
  22. 2024DIVERSE: A Dataset of YouTube Video Comment Stances with a Data Programming Modelcited by 2
  23. 2024Automatic Classification of News Subjects in Broadcast News: Application to a Gender Bias Representation Analysiscited by 2
  24. 2026From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Modelscited by 2
  25. 2023An Adaptive Method for Weak Supervision with Drifting Datacited by 1
  26. 2023Relation Extraction in underexplored biomedical domains: A diversity-optimised sampling and synthetic data generation approachcited by 1
  27. 2024Leveraging Large Language Models for Structure Learning in Prompted Weak Supervisioncited by 1
  28. 2024ActiveDP: Bridging Active Learning and Data Programmingcited by 1
  29. 2024Distilling Named Entity Recognition Models for Endangered Species from Large Language Modelscited by 1
  30. 2024WeShap: Weak Supervision Source Evaluation with Shapley Valuescited by 1
  31. 2024Automatic identification of diagnosis from hospital discharge letters via weakly supervised Natural Language Processingcited by 1
  32. 2025MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detectioncited by 1
  33. 2026Structured Exploration and Exploitation of Label Functions for Automated Data Annotationcited by 1
  34. 2022Regularized Data Programming with Automated Bayesian Prior Selectioncited by 0
  35. 2023Local Boosting for Weakly-Supervised Learningcited by 0
  36. 2025ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classificationcited by 0
  37. 2026MemeScouts@LT-EDI 2026: Asking the Right Questions -- Prompted Weak Supervision for Meme Hate Speech Detectioncited by 0
  38. 2026RuleChef: Grounding LLM Task Knowledge in Human-Editable Rulescited by 0

50 most cited of 52 All 52 (JSON)