# Open, Closed, or Small Language Models for Text Classification?

arXiv: https://arxiv.org/abs/2308.10092 · HTML: https://mlsanity.com/paper/2308.10092/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Portuguese Named Entity Recognition using BERT-CRF](https://mlsanity.com/paper/1909.10649/) (2019)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4](https://mlsanity.com/paper/2305.14928/) (2023)
- [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023)
- [TIMME: Twitter Ideology-detection via Multi-task Multi-relational Embedding](https://mlsanity.com/paper/2006.01321/) (2020)
- [Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign](https://mlsanity.com/paper/1802.04291/) (2018)
- [The Surprising Performance of Simple Baselines for Misinformation Detection](https://mlsanity.com/paper/2104.06952/) (2021)
- [Red Bots Do It Better: Comparative Analysis of Social Bot Partisan Behavior](https://mlsanity.com/paper/1902.02765/) (2019)
- [Ideology Detection for Twitter Users with Heterogeneous Types of Links](https://mlsanity.com/paper/1612.08207/) (2016)
- [Application of Pre-training Models in Named Entity Recognition](https://mlsanity.com/paper/2002.08902/) (2020)

## Built on by (most cited first, 30 of 34)

- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Best Practices for Text Annotation with Large Language Models](https://mlsanity.com/paper/2402.05129/) (2024)
- [The Open Source Advantage in Large Language Models (LLMs)](https://mlsanity.com/paper/2412.12004/) (2024)
- [Web Retrieval Agents for Evidence-Based Misinformation Detection](https://mlsanity.com/paper/2409.00009/) (2024)
- [GPTs Are Multilingual Annotators for Sequence Generation Tasks](https://mlsanity.com/paper/2402.05512/) (2024)
- [Automated Text Scoring in the Age of Generative AI for the GPU-poor](https://mlsanity.com/paper/2407.01873/) (2024)
- [A Guide to Misinformation Detection Data and Evaluation](https://mlsanity.com/paper/2411.05060/) (2024)
- [Are Chatbots Reliable Text Annotators? Sometimes](https://mlsanity.com/paper/2311.05769/) (2023)
- [Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigation](https://mlsanity.com/paper/2401.06920/) (2024)
- [ScholaWrite: A Dataset of End-to-End Scholarly Writing Process](https://mlsanity.com/paper/2502.02904/) (2025)
- [Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators](https://mlsanity.com/paper/2411.05854/) (2024)
- [Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning](https://mlsanity.com/paper/2412.08587/) (2024)
- [APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT](https://mlsanity.com/paper/2402.01697/) (2024)
- [Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detection](https://mlsanity.com/paper/2506.06347/) (2025)
- [Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks](https://mlsanity.com/paper/2402.10597/) (2024)
- [Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models](https://mlsanity.com/paper/2404.17985/) (2024)
- [LlamBERT: Large-scale low-cost data annotation in NLP](https://mlsanity.com/paper/2403.15938/) (2024)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [Leveraging Open-Source Large Language Models for Native Language Identification](https://mlsanity.com/paper/2409.09659/) (2024)
- [Data-Centric AI Governance: Addressing the Limitations of Model-Focused Policies](https://mlsanity.com/paper/2409.17216/) (2024)
- [Your Extreme Multi-label Classifier is Secretly a Hierarchical Text Classifier for Free](https://mlsanity.com/paper/2411.13687/) (2024)
- [ScamSpot: Fighting Financial Fraud in Instagram Comments](https://mlsanity.com/paper/2402.08869/) (2024)
- [Illuminating Blind Spots of Language Models with Targeted Agent-in-the-Loop Synthetic Data](https://mlsanity.com/paper/2403.17860/) (2024)
- [Suicide Risk Assessment on Social Media with Semi-Supervised Learning](https://mlsanity.com/paper/2411.12767/) (2024)
- [Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneck](https://mlsanity.com/paper/2412.08528/) (2024)
- [Rethinking the Understanding Ability across LLMs through Mutual Information](https://mlsanity.com/paper/2505.23790/) (2025)
- [Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models](https://mlsanity.com/paper/2506.07424/) (2025)
- [Measuring Negative Campaigning across Languages with Large Language Models: A Study of 18 Million Tweets in 19 Countries](https://mlsanity.com/paper/2507.17636/) (2025)
- [What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems](https://mlsanity.com/paper/2605.20548/) (2026)
- [Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data](https://mlsanity.com/paper/2401.12295/) (2024)

## Abstract (arXiv)

Recent advancements in large language models have demonstrated remarkable capabilities across various NLP tasks. But many questions remain, including whether open-source models match closed ones, why these models excel or struggle with certain tasks, and what types of practical procedures can improve performance. We address these questions in the context of classification by evaluating three classes of models using eight datasets across three distinct tasks: named entity recognition, political party prediction, and misinformation detection. While larger LLMs often lead to improved performance, open-source models can rival their closed-source counterparts by fine-tuning. Moreover, supervised smaller models, like RoBERTa, can achieve similar or even greater performance in many datasets compared to generative LLMs. On the other hand, closed models maintain an advantage in hard tasks that demand the most generalizability. This study underscores the importance of model selection based on task requirements