mlsanityData to 14 Oct 2026

Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification

arXiv 2404.11122 ↗First posted 17 Apr 2024cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

This study is part of the debate on the efficiency of large versus small language models for text classification by prompting.We assess the performance of small language models in zero-shot text classification, challenging the prevailing dominance of large models.Across 15 datasets, our investigation benchmarks language models from 77M to 40B parameters using different architectures and scoring functions. Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts.We developed and shared a comprehensive open-source repository that encapsulates our methodologies. This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

26 of 50 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
14 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  4. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  5. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  6. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  7. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  8. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  9. 2018Zero-shot User Intent Detection via Capsule Neural Networkscited by 54
  10. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 52
  11. 2023Explanation-based Finetuning Makes Models More Robust to Spurious Cuescited by 25
  12. 2022What Makes Pre-trained Language Models Better Zero-shot Learners?cited by 23
  13. 2022Pre-trained Language Models Can be Fully Zero-Shot Learnerscited by 22
  14. 2022Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representationscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 6.

18 papers.

Sort:

By area:

  1. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  2. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 88
  3. 2024InkubaLM: A small language model for low-resource African languagescited by 18
  4. 2025Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfercited by 3
  5. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  6. 2024Chain-of-MetaWriting: Linguistic and Textual Analysis of How Small Language Models Write Young Students Textscited by 2
  7. 2025Integrating Language Models for Enhanced Network State Monitoring in DRL-Based SFC Provisioningcited by 2
  8. 2025Small Language Models in the Real World: Insights from Industrial Text Classificationcited by 2
  9. 2025Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Modelscited by 1
  10. 2025Multi-stage Prompt Refinement for Mitigating Hallucinations in Large Language Modelscited by 1
  11. 2026SC-MAS: Constructing Cost-Efficient Multi-Agent Systems with Edge-Level Heterogeneous Collaborationcited by 1
  12. 2026BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMscited by 1
6 more
  1. 2026EARS: Explanatory Abstention for Reliable Sub-Agent Modeling in Large-scale Multi-Agent Systemscited by 1
  2. 2024Personal Intelligence System UniLM: Hybrid On-Device Small Language Model and Server-Based Large Language Model for Malay Nusantaracited by 0
  3. 2024Adaptable Embeddings Network (AEN)cited by 0
  4. 2026Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?cited by 0
  5. 2026SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approachcited by 0
  6. 2026Zero-Shot Visualization: Exploring Text Corpora with User-Prompted Axescited by 0