# Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches

arXiv: https://arxiv.org/abs/2211.16285 · HTML: https://mlsanity.com/paper/2211.16285/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021)
- [A Decade of Knowledge Graphs in Natural Language Processing: A Survey](https://mlsanity.com/paper/2210.00105/) (2022)
- [Does Putting a Linguist in the Loop Improve NLU Data Collection?](https://mlsanity.com/paper/2104.07179/) (2021)
- [Integrating Semantic Knowledge to Tackle Zero-shot Text Classification](https://mlsanity.com/paper/1903.12626/) (2019)
- [Train Once, Test Anywhere: Zero-Shot Learning for Text Classification](https://mlsanity.com/paper/1712.05972/) (2017)
- [PatternRank: Leveraging Pretrained Language Models and Part of Speech for Unsupervised Keyphrase Extraction](https://mlsanity.com/paper/2210.05245/) (2022)
- [Lbl2Vec: An Embedding-Based Approach for Unsupervised Document Retrieval on Predefined Topics](https://mlsanity.com/paper/2210.06023/) (2022)
- [Joint Embedding of Hierarchical Categories and Entities for Concept Categorization and Dataless Classification](https://mlsanity.com/paper/1607.07956/) (2016)
- [DocSCAN: Unsupervised Text Classification via Learning from Neighbors](https://mlsanity.com/paper/2105.04024/) (2021)

## Built on by (most cited first, 30 of 39)

- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Differentiate ChatGPT-generated and Human-written Medical Texts](https://mlsanity.com/paper/2304.11567/) (2023)
- [Automatic Evaluation of Healthcare LLMs Beyond Question-Answering](https://mlsanity.com/paper/2502.06666/) (2025)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models](https://mlsanity.com/paper/2311.15614/) (2023)
- [Towards Multi-modal Transformers in Federated Learning](https://mlsanity.com/paper/2404.12467/) (2024)
- [Interpretable Text Embeddings and Text Similarity Explanation: A Survey](https://mlsanity.com/paper/2502.14862/) (2025)
- [The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators](https://mlsanity.com/paper/2407.11004/) (2024)
- [Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data](https://mlsanity.com/paper/2411.08028/) (2024)
- [FLEXITOKENS: Flexible Tokenization for Evolving Language Models](https://mlsanity.com/paper/2507.12720/) (2025)
- [Exploring the Landscape of Natural Language Processing Research](https://mlsanity.com/paper/2307.10652/) (2023)
- [Approximating Human-Like Few-shot Learning with GPT-based Compression](https://mlsanity.com/paper/2308.06942/) (2023)
- [Towards Achieving Concept Completeness for Textual Concept Bottleneck Models](https://mlsanity.com/paper/2502.11100/) (2025)
- [Efficient Domain Adaptation of Sentence Embeddings Using Adapters](https://mlsanity.com/paper/2307.03104/) (2023)
- [Can Large Language Models Design Accurate Label Functions?](https://mlsanity.com/paper/2311.00739/) (2023)
- [AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledge](https://mlsanity.com/paper/2307.07851/) (2023)
- [Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation](https://mlsanity.com/paper/2506.03857/) (2025)
- [Retrieval Augmented Zero-Shot Text Classification](https://mlsanity.com/paper/2406.15241/) (2024)
- [MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Models](https://mlsanity.com/paper/2407.09972/) (2024)
- [WeShap: Weak Supervision Source Evaluation with Shapley Values](https://mlsanity.com/paper/2406.11010/) (2024)
- [SpMis: An Investigation of Synthetic Spoken Misinformation Detection](https://mlsanity.com/paper/2409.11308/) (2024)
- [Medifact at PerAnsSumm 2025: Leveraging Lightweight Models for Perspective-Specific Summarization of Clinical Q&A Forums](https://mlsanity.com/paper/2503.16513/) (2025)
- [Learning over von Mises-Fisher Distributions via a Wasserstein-like Geometry](https://mlsanity.com/paper/2504.14164/) (2025)
- [Enterprise Use Cases Combining Knowledge Graphs and Natural Language Processing](https://mlsanity.com/paper/2404.01443/) (2024)
- [Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?](https://mlsanity.com/paper/2407.12626/) (2024)
- [Hybrid Student-Teacher Large Language Model Refinement for Cancer Toxicity Symptom Extraction](https://mlsanity.com/paper/2408.04775/) (2024)
- [Generative AI-Powered Plugin for Robust Federated Learning in Heterogeneous IoT Networks](https://mlsanity.com/paper/2410.23824/) (2024)
- [Enhancing Multi-Class Disease Classification: Neoplasms, Cardiovascular, Nervous System, and Digestive Disorders Using Advanced LLMs](https://mlsanity.com/paper/2411.12712/) (2024)
- [Learning Predictive Checklists with Probabilistic Logic Programming](https://mlsanity.com/paper/2411.16790/) (2024)
- [FHBench: Towards Efficient and Personalized Federated Learning for Multimodal Healthcare](https://mlsanity.com/paper/2504.10817/) (2025)

## Abstract (arXiv)

Text classification of unseen classes is a challenging Natural Language Processing task and is mainly attempted using two different types of approaches. Similarity-based approaches attempt to classify instances based on similarities between text document representations and class description representations. Zero-shot text classification approaches aim to generalize knowledge gained from a training task by assigning appropriate labels of unknown classes to text documents. Although existing studies have already investigated individual approaches to these categories, the experiments in literature do not provide a consistent comparison. This paper addresses this gap by conducting a systematic evaluation of different similarity-based and zero-shot approaches for text classification of unseen classes. Different state-of-the-art approaches are benchmarked on four text classification datasets, including a new dataset from the medical domain. Additionally, novel SimCSE and SBERT-based baselines are proposed, as other baselines used in existing work yield weak classification results and are easily outperformed. Finally, the novel similarity-based Lbl2TransformerVec approach is presented, which outperforms previous state-of-the-art approaches in unsupervised text classification. Our experiments show that similarity-based approaches significantly outperform zero-shot approaches in most cases. Additionally, using SimCSE or SBERT embeddings instead of simpler text representations increases similarity-based classification results even further.