# Building Efficient Universal Classifiers with Natural Language Inference

arXiv: https://arxiv.org/abs/2312.17543 · HTML: https://mlsanity.com/paper/2312.17543/

## Key facts

- Cited by 19 papers in the map (data through 7 Oct 2026).
- Builds on 38 of its 73 references that are in the map.
- First posted on arXiv 29 Dec 2023.

## Builds on (most cited first, 30 of 38)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021)
- [The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI](https://mlsanity.com/paper/2310.16787/) (2023)

## Built on by (most cited first, 19 of 19)

- [Belief Revision: The Adaptability of Large Language Models Reasoning](https://mlsanity.com/paper/2406.19764/) (2024)
- [DiffGuard: Text-Based Safety Checker for Diffusion Models](https://mlsanity.com/paper/2412.00064/) (2024)
- [GLiClass: Generalist Lightweight Model for Sequence Classification Tasks](https://mlsanity.com/paper/2508.07662/) (2025)
- [TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generation](https://mlsanity.com/paper/2504.18535/) (2025)
- [Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text](https://mlsanity.com/paper/2409.02078/) (2024)
- [Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice](https://mlsanity.com/paper/2409.07713/) (2024)
- [Representation Consistency for Accurate and Coherent LLM Answer Aggregation](https://mlsanity.com/paper/2506.21590/) (2025)
- [SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?](https://mlsanity.com/paper/2602.00327/) (2026)
- [BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs](https://mlsanity.com/paper/2603.11991/) (2026)
- [Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering](https://mlsanity.com/paper/2604.07985/) (2026)
- [DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation](https://mlsanity.com/paper/2606.31537/) (2026)
- [LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioning](https://mlsanity.com/paper/2509.22211/) (2025)
- [Cross-Stage Coherence in Hierarchical Driving VQA: Explicit Baselines and Learned Gated Context Projectors](https://mlsanity.com/paper/2604.22560/) (2026)
- [The Consistency Illusion: How Multi-Agent Debate Hides Reasoning Misalignment](https://mlsanity.com/paper/2606.08457/) (2026)
- [Population-Level Profiling of DSM-5 Depressive Symptoms Among Self-Reported ADHD and ASD Users on Twitter: An Exploratory Study Using Advanced NLP and Statistical Analysis](https://mlsanity.com/paper/2607.05626/) (2026)
- [Can Artificial Intelligence Support Healthcare and Mental Health Through Early Cyberbullying Detection ? The Impact of Emotion-Aware AI on Proactive Online Safety](https://mlsanity.com/paper/2609.09735/) (2026)
- [Typed Decision Models: An Early Evidence Audit and Evaluation Checklist](https://mlsanity.com/paper/2609.32160/) (2026)
- [Chinese-Jev: Bringing System One Model to Chinese-Language Tasks](https://mlsanity.com/paper/2609.36965/) (2026)
- [A Comparative Explainability Framework for DeBERTa-v3 in Zero-Shot Medical Abstract Classification](https://mlsanity.com/paper/2610.02116/) (2026)

## Abstract (arXiv)

Generative Large Language Models (LLMs) have become the mainstream choice for fewshot and zeroshot learning thanks to the universality of text generation. Many users, however, do not need the broad capabilities of generative LLMs when they only want to automate a classification task. Smaller BERT-like models can also learn universal tasks, which allow them to do any text classification task without requiring fine-tuning (zeroshot classification) or to learn new tasks with only a few examples (fewshot), while being significantly more efficient than generative LLMs. This paper (1) explains how Natural Language Inference (NLI) can be used as a universal classification task that follows similar principles as instruction fine-tuning of generative LLMs, (2) provides a step-by-step guide with reusable Jupyter notebooks for building a universal classifier, and (3) shares the resulting universal classifier that is trained on 33 datasets with 389 diverse classes. Parts of the code we share has been used to train our older zeroshot classifiers that have been downloaded more than 55 million times via the Hugging Face Hub as of December 2023. Our new classifier improves zeroshot performance by 9.4%.