# A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches arXiv: https://arxiv.org/abs/2305.12749 ยท HTML: https://mlsanity.com/paper/2305.12749/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 44 references that are in the map. - First posted on arXiv 22 May 2023. ## Builds on (most cited first, 30 of 30) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015) - [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021) - [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021) - [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020) - [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021) - [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020) - [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021) - [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019) - [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021) - [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021) - [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021) - [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020) - [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021) - [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020) - [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017) - [Weakly-Supervised Neural Text Classification](https://mlsanity.com/paper/1809.01478/) (2018) - [Prototypical Calibration for Few-shot Learning of Language Models](https://mlsanity.com/paper/2205.10183/) (2022) - [X-Class: Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2010.12794/) (2020) - [Discriminative Topic Mining via Category-Name Guided Text Embedding](https://mlsanity.com/paper/1908.07162/) (2019) - [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022) - [Weakly-supervised Text Classification Based on Keyword Graph](https://mlsanity.com/paper/2110.02591/) (2021) - [Pre-trained Language Models Can be Fully Zero-Shot Learners](https://mlsanity.com/paper/2212.06950/) (2022) - [AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels](https://mlsanity.com/paper/2208.14362/) (2022) - [LIME: Weakly-Supervised Text Classification Without Seeds](https://mlsanity.com/paper/2210.06720/) (2022) ## Built on by (most cited first, 6 of 6) - [Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training](https://mlsanity.com/paper/2306.07193/) (2023) - [Incubating Text Classifiers Following User Instruction with Nothing but LLM](https://mlsanity.com/paper/2404.10877/) (2024) - [Less than One-shot: Named Entity Recognition via Extremely Weak Supervision](https://mlsanity.com/paper/2311.02861/) (2023) - [RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules](https://mlsanity.com/paper/2403.02932/) (2024) - [Text Grafting: Near-Distribution Weak Supervision for Minority Classes in Text Classification](https://mlsanity.com/paper/2406.11115/) (2024) - [Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank](https://mlsanity.com/paper/2410.23066/) (2024) ## Abstract (arXiv) Etremely Weakly Supervised Text Classification (XWS-TC) refers to text classification based on minimal high-level human guidance, such as a few label-indicative seed words or classification instructions. There are two mainstream approaches for XWS-TC, however, never being rigorously compared: (1) training classifiers based on pseudo-labels generated by (softly) matching seed words (SEED) and (2) prompting (and calibrating) language models using classification instruction (and raw texts) to decode label words (PROMPT). This paper presents the first XWS-TC benchmark to compare the two approaches on fair grounds, where the datasets, supervisions, and hyperparameter choices are standardized across methods. Our benchmarking results suggest that (1) Both SEED and PROMPT approaches are competitive and there is no clear winner; (2) SEED is empirically more tolerant than PROMPT to human guidance (e.g., seed words, classification instructions, and label words) changes; (3) SEED is empirically more selective than PROMPT to the pre-trained language models; (4) Recent SEED and PROMPT methods have close connections and a clustering post-processing step based on raw in-domain texts is a strong performance booster to both. We hope this benchmark serves as a guideline in selecting XWS-TC methods in different scenarios and stimulate interest in developing guidance- and model-robust XWS-TC methods. We release the repo at https://github.com/ZihanWangKi/x-TC.