# XPASC: Measuring Generalization in Weak Supervision by Explainability and Association

arXiv: https://arxiv.org/abs/2206.01444 · HTML: https://mlsanity.com/paper/2206.01444/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 3 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Training Complex Models with Multi-Task Weak Supervision](https://mlsanity.com/paper/1810.02840/) (2018)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach](https://mlsanity.com/paper/2010.07835/) (2020)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Learning from Rules Generalizing Labeled Exemplars](https://mlsanity.com/paper/2004.06025/) (2020)
- [Denoising Multi-Source Weak Supervision for Neural Text Classification](https://mlsanity.com/paper/2010.04582/) (2020)
- [How benign is benign overfitting?](https://mlsanity.com/paper/2007.04028/) (2020)
- [End-to-End Weak Supervision](https://mlsanity.com/paper/2107.02233/) (2021)
- [Overfitting Mechanism and Avoidance in Deep Neural Networks](https://mlsanity.com/paper/1901.06566/) (2019)
- [Strength from Weakness: Fast Learning Using Weak Supervision](https://mlsanity.com/paper/2002.08483/) (2020)
- [Considering Likelihood in NLP Classification Explanations with Occlusion and Language Modeling](https://mlsanity.com/paper/2004.09890/) (2020)
- [Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming](https://mlsanity.com/paper/2203.01382/) (2022)
- [Understanding Programmatic Weak Supervision via Source-aware Influence Function](https://mlsanity.com/paper/2205.12879/) (2022)
- [Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries](https://mlsanity.com/paper/2005.00524/) (2020)
- [Knodle: Modular Weakly Supervised Learning with PyTorch](https://mlsanity.com/paper/2104.11557/) (2021)
- [Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision](https://mlsanity.com/paper/1711.00313/) (2017)
- [KnowMAN: Weakly Supervised Multinomial Adversarial Networks](https://mlsanity.com/paper/2109.07994/) (2021)

## Built on by (most cited first, 1 of 1)

- [Learning Stable Predictors from Weak Supervision under Distribution Shift](https://mlsanity.com/paper/2604.05002/) (2026)

## Abstract (arXiv)

Weak supervision is leveraged in a wide range of domains and tasks due to its ability to create massive amounts of labeled data, requiring only little manual effort. Standard approaches use labeling functions to specify signals that are relevant for the labeling. It has been conjectured that weakly supervised models over-rely on those signals and as a result suffer from overfitting. To verify this assumption, we introduce a novel method, XPASC (eXPlainability-Association SCore), for measuring the generalization of a model trained with a weakly supervised dataset. Considering the occurrences of features, classes and labeling functions in a dataset, XPASC takes into account the relevance of each feature for the predictions of the model as well as the associations of the feature with the class and the labeling function, respectively. The association in XPASC can be measured in two variants: XPASC-CHI SQAURE measures associations relative to their statistical significance, while XPASC-PPMI measures association strength more generally. We use XPASC to analyze KnowMAN, an adversarial architecture intended to control the degree of generalization from the labeling functions and thus to mitigate the problem of overfitting. On one hand, we show that KnowMAN is able to control the degree of generalization through a hyperparameter. On the other hand, results and qualitative analysis show that generalization and performance do not relate one-to-one, and that the highest degree of generalization does not necessarily imply the best performance. Therefore methods that allow for controlling the amount of generalization can achieve the right degree of benign overfitting. Our contributions in this study are i) the XPASC score to measure generalization in weakly-supervised models, ii) evaluation of XPASC across datasets and models and iii) the release of the XPASC implementation.