mlsanityData to 7 Oct 2026

Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification

arXiv 2406.08660 ↗First posted 12 Jun 2024cs.CL cs.AIcited by 66

Not on the map. Why?

Abstract (arXiv)

Generative AI offers a simple, prompt-based alternative to fine-tuning smaller BERT-style LLMs for text classification tasks. This promises to eliminate the need for manually labeled training data and task-specific model training. However, it remains an open question whether tools like ChatGPT can deliver on this promise. In this paper, we show that smaller, fine-tuned LLMs (still) consistently and significantly outperform larger, zero-shot prompted models in text classification. We compare three major generative AI models (ChatGPT with GPT-3.5/GPT-4 and Claude Opus) with several fine-tuned LLMs across a diverse set of classification tasks (sentiment, approval/disapproval, emotions, party positions) and text categories (news, tweets, speeches). We find that fine-tuning with application-specific training data achieves superior performance in all cases. To make this approach more accessible to a broader audience, we provide an easy-to-use toolkit alongside this paper. Our toolkit, accompanied by non-technical step-by-step guidance, enables users to select and fine-tune BERT-like LLMs for any classification task with minimal technical and computational effort.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

23 of 46 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
11 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  5. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  6. 2023Textbooks Are All You Needcited by 561
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  8. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  9. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  10. 2022Broken Neural Scaling Lawscited by 102
  11. 2024Language Models for Text Classification: Is In-Context Learning Enough?cited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 28; 2026: 24.

66 papers.

Sort:

By area:

  1. 2025Detecting High-Stakes Interactions with Activation Probescited by 45
  2. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  3. 2025FinSage: A Multi-aspect RAG System for Financial Filings Question Answeringcited by 18
  4. 2024Enabling Novel Mission Operations and Interactions with ROSA: The Robot Operating System Agentcited by 16
  5. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  6. 2024ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Taskscited by 8
  7. 2024Polish-English medical knowledge transfer: A new benchmark and resultscited by 8
  8. 2024Practical Considerations for Agentic LLM Systemscited by 8
  9. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
  10. 2025Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classificationcited by 7
  11. 2024Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Researchcited by 6
  12. 2025SAMChat: Introducing Chain of Thought Reasoning and GRPO to a Multimodal Small Language Model for Small Scale Remote Sensingcited by 6
38 more
  1. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5
  2. 2024Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Datacited by 4
  3. 2024Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Sciencecited by 4
  4. 2025Towards Robust Few-Shot Text Classification Using Transformer Architectures and Dual Loss Strategiescited by 4
  5. 2026The Embedder's Dilemma: LLMs Are Better, but at What Cost?cited by 4
  6. 2024Arctic-TILT. Business Document Understanding at Sub-Billion Scalecited by 3
  7. 2025Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Taskscited by 3
  8. 2025Few-Shot Vision-Language Reasoning for Satellite Imagery via Verifiable Rewardscited by 3
  9. 2024Leveraging Open-Source Large Language Models for Native Language Identificationcited by 2
  10. 2025ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reportscited by 2
  11. 2025Deep literature reviews: an application of fine-tuned language models to migration researchcited by 2
  12. 2025Rethinking Hate Speech Detection on Social Media: Can LLMs Replace Traditional Models?cited by 2
  13. 2025A Scalable and Efficient Signal Integration System for Job Matchingcited by 2
  14. 2025How and Where to Translate? The Impact of Translation Strategies in Cross-lingual LLM Promptingcited by 2
  15. 2024Data Quality Enhancement on the Basis of Diversity with Large Language Models for Text Classification: Uncovered, Difficult, and Noisycited by 1
  16. 2025Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQscited by 1
  17. 2025Evaluating Compact LLMs for Zero-Shot Iberian Language Tasks on End-User Devicescited by 1
  18. 2025Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learningcited by 1
  19. 2025A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Healthcited by 1
  20. 2025Prompting Strategies for Language Model-Based Item Generation in K-12 Education: Bridging the Gap Between Small and Large Language Modelscited by 1
  21. 2025Multi-Lingual Implicit Discourse Relation Recognition with Multi-Label Hierarchical Learningcited by 1
  22. 2025SALSA: Single-pass Autoregressive LLM Structured Classificationcited by 1
  23. 2025Efficient Text Classification with Conformal In-Context Learningcited by 1
  24. 2026Cost-Aware Model Selection for Text Classification: Multi-Objective Trade-offs Between Fine-Tuned Encoders and LLM Prompting in Productioncited by 1
  25. 2026Enhancing Linux Privilege Escalation Attack Capabilities of Local LLM Agentscited by 1
  26. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioningcited by 1
  27. 2024Combining Autoregressive and Autoencoder Language Models for Text Classificationcited by 0
  28. 2025AdaPhish: AI-Powered Adaptive Defense and Education Resource Against Deceptive Emailscited by 0
  29. 2025Agentic AI for Scaling Diagnosis and Care in Neurodegenerative Diseasecited by 0
  30. 2025Evaluating Apple Intelligence's Writing Tools for Privacy Against Large Language Model-Based Inference Attacks: Insights from Early Datasetscited by 0
  31. 2025DS@GT at CheckThat! 2025: Ensemble Methods for Detection of Scientific Discourse on Social Mediacited by 0
  32. 2025Trained Miniatures: Low cost, High Efficacy SLMs for Sales & Marketingcited by 0
  33. 2025PEARL: Peer-Enhanced Adaptive Radio via On-Device LLMcited by 0
  34. 2025Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacementcited by 0
  35. 2026CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Modelscited by 0
  36. 2026Can Large Language Models Make Everyone Happy?cited by 0
  37. 2026PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classificationcited by 0
  38. 2026DLT-Corpus: A Large-Scale Text Collection for the Distributed Ledger Technology Domaincited by 0

50 most cited of 66 All 66 (JSON)