# Weakly Supervised Text Classification using Supervision Signals from a Language Model

arXiv: https://arxiv.org/abs/2205.06604 · HTML: https://mlsanity.com/paper/2205.06604/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018)
- [Zero-shot User Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1809.00385/) (2018)
- [X-Class: Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2010.12794/) (2020)
- [Weakly-Supervised Hierarchical Text Classification](https://mlsanity.com/paper/1812.11270/) (2018)
- [Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling](https://mlsanity.com/paper/1611.06639/) (2016)
- [Integrating Semantic Knowledge to Tackle Zero-shot Text Classification](https://mlsanity.com/paper/1903.12626/) (2019)
- [Variational Knowledge Graph Reasoning](https://mlsanity.com/paper/1803.06581/) (2018)
- [A Variational Approach to Weakly Supervised Document-Level Multi-Aspect Sentiment Classification](https://mlsanity.com/paper/1904.05055/) (2019)
- [Unsupervised Induction of Semantic Roles within a Reconstruction-Error Minimization Framework](https://mlsanity.com/paper/1412.2812/) (2014)
- [Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders](https://mlsanity.com/paper/1603.09128/) (2016)
- [Relation Discovery with Out-of-Relation Knowledge Base as Supervision](https://mlsanity.com/paper/1905.01959/) (2019)

## Built on by (most cited first, 10 of 10)

- [TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision](https://mlsanity.com/paper/2403.00165/) (2024)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training](https://mlsanity.com/paper/2305.13723/) (2023)
- [MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities](https://mlsanity.com/paper/2304.01969/) (2023)
- [XAI-CLASS: Explanation-Enhanced Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2311.00189/) (2023)
- [RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules](https://mlsanity.com/paper/2403.02932/) (2024)
- [On-the-fly Denoising for Data Augmentation in Natural Language Understanding](https://mlsanity.com/paper/2212.10558/) (2022)
- [Substituting Data Annotation with Balanced Updates and Collective Loss in Multi-label Text Classification](https://mlsanity.com/paper/2309.13543/) (2023)
- [A Consensus-Driven Multi-LLM Pipeline for Missing-Person Investigations](https://mlsanity.com/paper/2603.08954/) (2026)
- [LLM-based Schema-Guided Extraction and Validation of Missing-Person Intelligence from Heterogeneous Data Sources](https://mlsanity.com/paper/2604.06571/) (2026)

## Abstract (arXiv)

Solving text classification in a weakly supervised manner is important for real-world applications where human annotations are scarce. In this paper, we propose to query a masked language model with cloze style prompts to obtain supervision signals. We design a prompt which combines the document itself and "this article is talking about [MASK]." A masked language model can generate words for the [MASK] token. The generated words which summarize the content of a document can be utilized as supervision signals. We propose a latent variable model to learn a word distribution learner which associates generated words to pre-defined categories and a document classifier simultaneously without using any annotated data. Evaluation on three datasets, AGNews, 20Newsgroups, and UCINews, shows that our method can outperform baselines by 2%, 4%, and 3%.