# ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision

arXiv: https://arxiv.org/abs/2204.06863 · HTML: https://mlsanity.com/paper/2204.06863/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 71 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Training Complex Models with Multi-Task Weak Supervision](https://mlsanity.com/paper/1810.02840/) (2018)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [Fine-Grained Spoiler Detection from Large-Scale Review Corpora](https://mlsanity.com/paper/1905.13416/) (2019)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Learning from Rules Generalizing Labeled Exemplars](https://mlsanity.com/paper/2004.06025/) (2020)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling](https://mlsanity.com/paper/2012.06046/) (2020)
- [Self-Training with Weak Supervision](https://mlsanity.com/paper/2104.05514/) (2021)
- [Denoising Multi-Source Weak Supervision for Neural Text Classification](https://mlsanity.com/paper/2010.04582/) (2020)
- [End-to-End Weak Supervision](https://mlsanity.com/paper/2107.02233/) (2021)
- [Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News](https://mlsanity.com/paper/2004.01732/) (2020)
- [Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling](https://mlsanity.com/paper/1910.04289/) (2019)
- [Ontology-driven weak supervision for clinical entity classification in electronic health records](https://mlsanity.com/paper/2008.01972/) (2020)
- [Semi-Supervised Data Programming with Subset Selection](https://mlsanity.com/paper/2008.09887/) (2020)
- [Training a Neural Network in a Low-Resource Setting on Automatically Annotated Noisy Data](https://mlsanity.com/paper/1807.00745/) (2018)
- [Weak Supervision and Referring Attention for Temporal-Textual Association Learning](https://mlsanity.com/paper/2006.11747/) (2020)
- [ANEA: Distant Supervision for Low-Resource Named Entity Recognition](https://mlsanity.com/paper/2102.13129/) (2021)
- [Knodle: Modular Weakly Supervised Learning with PyTorch](https://mlsanity.com/paper/2104.11557/) (2021)
- [Distant Supervision for Relation Extraction with Linear Attenuation Simulation and Non-IID Relevance Embedding](https://mlsanity.com/paper/1812.09516/) (2018)
- [Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels](https://mlsanity.com/paper/1910.06061/) (2019)
- [Hybrid Supervision Learning for Pathology Whole Slide Image Classification](https://mlsanity.com/paper/2107.00934/) (2021)

## Built on by (most cited first, 1 of 1)

- [Refining Labeling Functions with Limited Labeled Data](https://mlsanity.com/paper/2505.23470/) (2025)

## Abstract (arXiv)

A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs), rule-based mechanisms that generate artificial labels for the associated classes. In this work, we investigate noise reduction techniques for WS based on the principle of k-fold cross-validation. We introduce a new algorithm ULF for Unsupervised Labeling Function correction, which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re-estimating this assignment on highly reliable cross-validated samples. Evaluation on multiple datasets confirms ULF's effectiveness in enhancing WS learning without the need for manual labeling.