# Knodle: Modular Weakly Supervised Learning with PyTorch

arXiv: https://arxiv.org/abs/2104.11557 · HTML: https://mlsanity.com/paper/2104.11557/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 67 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention](https://mlsanity.com/paper/1804.06987/) (2018)
- [Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning](https://mlsanity.com/paper/1805.09927/) (2018)
- [Learning Dependency Structures for Weak Supervision Models](https://mlsanity.com/paper/1903.05844/) (2019)
- [Learning with Weak Supervision for Email Intent Detection](https://mlsanity.com/paper/2005.13084/) (2020)
- [Analysing the Noise Model Error for Realistic Noisy Label Data](https://mlsanity.com/paper/2101.09763/) (2021)
- [Are Noisy Sentences Useless for Distant Supervised Relation Extraction?](https://mlsanity.com/paper/1911.09788/) (2019)
- [Identifying civilians killed by police with distantly supervised entity-event extraction](https://mlsanity.com/paper/1707.07086/) (2017)
- [Data Programming using Continuous and Quality-Guided Labeling Functions](https://mlsanity.com/paper/1911.09860/) (2019)
- [Improving Neural Relation Extraction with Positive and Unlabeled Learning](https://mlsanity.com/paper/1911.12556/) (2019)

## Built on by (most cited first, 7 of 7)

- [SepLL: Separating Latent Class Labels from Weak Supervision Noise](https://mlsanity.com/paper/2210.13898/) (2022)
- [Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal](https://mlsanity.com/paper/2306.04502/) (2023)
- [KnowMAN: Weakly Supervised Multinomial Adversarial Networks](https://mlsanity.com/paper/2109.07994/) (2021)
- [Label Augmentation with Reinforced Labeling for Weak Supervision](https://mlsanity.com/paper/2204.06436/) (2022)
- [ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision](https://mlsanity.com/paper/2204.06863/) (2022)
- [XPASC: Measuring Generalization in Weak Supervision by Explainability and Association](https://mlsanity.com/paper/2206.01444/) (2022)
- [WeaNF: Weak Supervision with Normalizing Flows](https://mlsanity.com/paper/2204.13409/) (2022)

## Abstract (arXiv)

Strategies for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task or integrated with a specific model architecture. In this work, we introduce Knodle, a software framework that treats weak data annotations, deep learning models, and methods for improving weakly supervised training as separate, modular components. This modularization gives the training process access to fine-grained information such as data set characteristics, matches of heuristic rules, or elements of the deep learning model ultimately used for prediction. Hence, our framework can encompass a wide range of training methods for improving weak supervision, ranging from methods that only look at correlations of rules and output classes (independently of the machine learning model trained with the resulting labels), to those that harness the interplay of neural networks and weakly labeled data. We illustrate the benchmarking potential of the framework with a performance comparison of several reference implementations on a selection of datasets that are already available in Knodle. The framework is published as an open-source Python package knodle and available at https://github.com/knodle/knodle.