# Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification

arXiv: https://arxiv.org/abs/2404.11122 · HTML: https://mlsanity.com/paper/2404.11122/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions](https://mlsanity.com/paper/2304.14402/) (2023)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [Zero-shot User Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1809.00385/) (2018)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Explanation-based Finetuning Makes Models More Robust to Spurious Cues](https://mlsanity.com/paper/2305.04990/) (2023)
- [What Makes Pre-trained Language Models Better Zero-shot Learners?](https://mlsanity.com/paper/2209.15206/) (2022)
- [Pre-trained Language Models Can be Fully Zero-Shot Learners](https://mlsanity.com/paper/2212.06950/) (2022)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)

## Built on by (most cited first, 18 of 18)

- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [MasRouter: Learning to Route LLMs for Multi-Agent Systems](https://mlsanity.com/paper/2502.11133/) (2025)
- [InkubaLM: A small language model for low-resource African languages](https://mlsanity.com/paper/2408.17024/) (2024)
- [Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfer](https://mlsanity.com/paper/2510.05846/) (2025)
- [Menta: A Small Language Model for On-Device Mental Health Prediction](https://mlsanity.com/paper/2512.02716/) (2025)
- [Chain-of-MetaWriting: Linguistic and Textual Analysis of How Small Language Models Write Young Students Texts](https://mlsanity.com/paper/2412.14986/) (2024)
- [Integrating Language Models for Enhanced Network State Monitoring in DRL-Based SFC Provisioning](https://mlsanity.com/paper/2502.11298/) (2025)
- [Small Language Models in the Real World: Insights from Industrial Text Classification](https://mlsanity.com/paper/2505.16078/) (2025)
- [Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models](https://mlsanity.com/paper/2506.07424/) (2025)
- [Multi-stage Prompt Refinement for Mitigating Hallucinations in Large Language Models](https://mlsanity.com/paper/2510.12032/) (2025)
- [SC-MAS: Constructing Cost-Efficient Multi-Agent Systems with Edge-Level Heterogeneous Collaboration](https://mlsanity.com/paper/2601.09434/) (2026)
- [BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs](https://mlsanity.com/paper/2603.11991/) (2026)
- [EARS: Explanatory Abstention for Reliable Sub-Agent Modeling in Large-scale Multi-Agent Systems](https://mlsanity.com/paper/2606.18668/) (2026)
- [Personal Intelligence System UniLM: Hybrid On-Device Small Language Model and Server-Based Large Language Model for Malay Nusantara](https://mlsanity.com/paper/2410.06973/) (2024)
- [Adaptable Embeddings Network (AEN)](https://mlsanity.com/paper/2411.13786/) (2024)
- [Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?](https://mlsanity.com/paper/2605.00964/) (2026)
- [SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach](https://mlsanity.com/paper/2608.00030/) (2026)
- [Zero-Shot Visualization: Exploring Text Corpora with User-Prompted Axes](https://mlsanity.com/paper/2610.06889/) (2026)

## Abstract (arXiv)

This study is part of the debate on the efficiency of large versus small language models for text classification by prompting.We assess the performance of small language models in zero-shot text classification, challenging the prevailing dominance of large models.Across 15 datasets, our investigation benchmarks language models from 77M to 40B parameters using different architectures and scoring functions. Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts.We developed and shared a comprehensive open-source repository that encapsulates our methodologies. This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.