# Language Models for Text Classification: Is In-Context Learning Enough?

arXiv: https://arxiv.org/abs/2403.17661 · HTML: https://mlsanity.com/paper/2403.17661/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 79 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)

## Built on by (most cited first, 30 of 40)

- [Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification](https://mlsanity.com/paper/2406.08660/) (2024)
- [RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content](https://mlsanity.com/paper/2406.11811/) (2024)
- [Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding](https://mlsanity.com/paper/2412.17837/) (2024)
- [ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks](https://mlsanity.com/paper/2507.01321/) (2025)
- [STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models](https://mlsanity.com/paper/2505.20645/) (2025)
- [Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning](https://mlsanity.com/paper/2412.08587/) (2024)
- [LLM-based Semantic Augmentation for Harmful Content Detection](https://mlsanity.com/paper/2504.15548/) (2025)
- [Active Few-Shot Learning for Text Classification](https://mlsanity.com/paper/2502.18782/) (2025)
- [Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understanding](https://mlsanity.com/paper/2510.04899/) (2025)
- [Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection](https://mlsanity.com/paper/2412.20595/) (2024)
- [IYKYK: Using language models to decode extremist cryptolects](https://mlsanity.com/paper/2506.05635/) (2025)
- [Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets](https://mlsanity.com/paper/2406.18239/) (2024)
- [Leveraging Open-Source Large Language Models for Native Language Identification](https://mlsanity.com/paper/2409.09659/) (2024)
- [Scalable Detection of Salient Entities in News Articles](https://mlsanity.com/paper/2405.20461/) (2024)
- [Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting](https://mlsanity.com/paper/2408.09365/) (2024)
- [Data Quality Enhancement on the Basis of Diversity with Large Language Models for Text Classification: Uncovered, Difficult, and Noisy](https://mlsanity.com/paper/2412.06575/) (2024)
- [Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages](https://mlsanity.com/paper/2505.20496/) (2025)
- [From Domain Documents to Requirements: Retrieval-Augmented Generation in the Space Industry](https://mlsanity.com/paper/2507.07689/) (2025)
- [A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health](https://mlsanity.com/paper/2508.05003/) (2025)
- [Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning](https://mlsanity.com/paper/2509.07768/) (2025)
- [Order from Chaos: Comparative Study of Ten Leading LLMs on Unstructured Data Categorization](https://mlsanity.com/paper/2510.13885/) (2025)
- [Efficient Text Classification with Conformal In-Context Learning](https://mlsanity.com/paper/2512.05732/) (2025)
- [CodeGuard: Improving LLM Guardrails in CS Education](https://mlsanity.com/paper/2602.02509/) (2026)
- [A Controlled Synthetic Benchmark for Educational Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2605.25502/) (2026)
- [Exploring Imbalanced Annotations for Effective In-Context Learning](https://mlsanity.com/paper/2502.04037/) (2025)
- [A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals](https://mlsanity.com/paper/2506.15208/) (2025)
- [Online Anti-sexist Speech: Identifying Resistance to Gender Bias in Political Discourse](https://mlsanity.com/paper/2508.11434/) (2025)
- [Trained Miniatures: Low cost, High Efficacy SLMs for Sales & Marketing](https://mlsanity.com/paper/2508.15617/) (2025)
- [From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models](https://mlsanity.com/paper/2509.09303/) (2025)
- [Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement](https://mlsanity.com/paper/2512.05525/) (2025)

## Abstract (arXiv)

Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches based on fine-tuning is the ability to understand instructions written in natural language (prompts), which helps them generalise better to different tasks and domains without the need for specific training data. This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances. However, existing research is limited in scale and lacks understanding of how text generation models combined with prompting techniques compare to more established methods for text classification such as fine-tuning masked language models. In this paper, we address this research gap by performing a large-scale evaluation study for 16 text classification datasets covering binary, multiclass, and multilabel problems. In particular, we compare zero- and few-shot approaches of large language models to fine-tuning smaller language models. We also analyse the results by prompt, classification type, domain, and number of labels. In general, the results show how fine-tuning smaller and more efficient language models can still outperform few-shot approaches of larger language models, which have room for improvement when it comes to text classification.