# WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categories

arXiv: https://arxiv.org/abs/2307.15293 · HTML: https://mlsanity.com/paper/2307.15293/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [HealthPrompt: A Zero-shot Learning Paradigm for Clinical Natural Language Processing](https://mlsanity.com/paper/2203.05061/) (2022)
- [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022)
- [Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation](https://mlsanity.com/paper/2201.08702/) (2022)
- [Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective](https://mlsanity.com/paper/2210.08590/) (2022)
- [Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models](https://mlsanity.com/paper/2210.14803/) (2022)
- [Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM](https://mlsanity.com/paper/2304.11872/) (2023)
- [Zero-Shot Text Classification via Self-Supervised Tuning](https://mlsanity.com/paper/2305.11442/) (2023)
- [Towards Open-Domain Topic Classification](https://mlsanity.com/paper/2306.17290/) (2023)

## Built on by (most cited first, 2 of 2)

- [LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods](https://mlsanity.com/paper/2407.00322/) (2024)
- [LOCUS: A System and Method for Low-Cost Customization for Universal Specialization](https://mlsanity.com/paper/2512.06239/) (2025)

## Abstract (arXiv)

Our research focuses on solving the zero-shot text classification problem in NLP, with a particular emphasis on innovative self-training strategies. To achieve this objective, we propose a novel self-training strategy that uses labels rather than text for training, significantly reducing the model's training time. Specifically, we use categories from Wikipedia as our training set and leverage the SBERT pre-trained model to establish positive correlations between pairs of categories within the same text, facilitating associative training. For new test datasets, we have improved the original self-training approach, eliminating the need for prior training and testing data from each target dataset. Instead, we adopt Wikipedia as a unified training dataset to better approximate the zero-shot scenario. This modification allows for rapid fine-tuning and inference across different datasets, greatly reducing the time required for self-training. Our experimental results demonstrate that this method can adapt the model to the target dataset within minutes. Compared to other BERT-based transformer models, our approach significantly reduces the amount of training data by training only on labels, not the actual text, and greatly improves training efficiency by utilizing a unified training set. Additionally, our method achieves state-of-the-art results on both the Yahoo Topic and AG News datasets.