# Conformal Predictor for Improving Zero-shot Text Classification Efficiency

arXiv: https://arxiv.org/abs/2210.12619 · HTML: https://mlsanity.com/paper/2210.12619/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [Zero-shot User Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1809.00385/) (2018)
- [PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models](https://mlsanity.com/paper/2204.01172/) (2022)
- [Generative and Discriminative Text Classification with Recurrent Neural Networks](https://mlsanity.com/paper/1703.01898/) (2017)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing](https://mlsanity.com/paper/2002.05829/) (2020)
- [Joint Embedding of Hierarchical Categories and Entities for Concept Categorization and Dataless Classification](https://mlsanity.com/paper/1607.07956/) (2016)

## Built on by (most cited first, 2 of 2)

- [Conformal Prediction for Natural Language Processing: A Survey](https://mlsanity.com/paper/2405.01976/) (2024)
- [Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://mlsanity.com/paper/2402.00707/) (2024)

## Abstract (arXiv)

Pre-trained language models (PLMs) have been shown effective for zero-shot (0shot) text classification. 0shot models based on natural language inference (NLI) and next sentence prediction (NSP) employ cross-encoder architecture and infer by making a forward pass through the model for each label-text pair separately. This increases the computational cost to make inferences linearly in the number of labels. In this work, we improve the efficiency of such cross-encoder-based 0shot models by restricting the number of likely labels using another fast base classifier-based conformal predictor (CP) calibrated on samples labeled by the 0shot model. Since a CP generates prediction sets with coverage guarantees, it reduces the number of target labels without excluding the most probable label based on the 0shot model. We experiment with three intent and two topic classification datasets. With a suitable CP for each dataset, we reduce the average inference time for NLI- and NSP-based models by 25.6% and 22.2% respectively, without dropping performance below the predefined error rate of 1%.