mlsanityData to 7 Oct 2026

Open, Closed, or Small Language Models for Text Classification?

arXiv 2308.10092 ↗First posted 19 Aug 2023cs.CL cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models have demonstrated remarkable capabilities across various NLP tasks. But many questions remain, including whether open-source models match closed ones, why these models excel or struggle with certain tasks, and what types of practical procedures can improve performance. We address these questions in the context of classification by evaluating three classes of models using eight datasets across three distinct tasks: named entity recognition, political party prediction, and misinformation detection. While larger LLMs often lead to improved performance, open-source models can rival their closed-source counterparts by fine-tuning. Moreover, supervised smaller models, like RoBERTa, can achieve similar or even greater performance in many datasets compared to generative LLMs. On the other hand, closed models maintain an advantage in hard tasks that demand the most generalizability. This study underscores the importance of model selection based on task requirements

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 24 of its 51 references that are in the map. First posted on arXiv 19 Aug 2023.

Builds on

24 of 51 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  9. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  10. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  11. 2017Fake News Detection on Social Media: A Data Mining Perspectivecited by 334
  12. 2023How is ChatGPT's behavior changing over time?cited by 265
12 more
  1. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  2. 2019A Survey on Recent Advances in Named Entity Recognition from Deep Learning modelscited by 117
  3. 2019Portuguese Named Entity Recognition using BERT-CRFcited by 73
  4. 2019CrossWeigh: Training Named Entity Tagger from Imperfect Annotationscited by 60
  5. 2023Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4cited by 45
  6. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  7. 2020TIMME: Twitter Ideology-detection via Multi-task Multi-relational Embeddingcited by 39
  8. 2018Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaigncited by 30
  9. 2021The Surprising Performance of Simple Baselines for Misinformation Detectioncited by 27
  10. 2019Red Bots Do It Better: Comparative Analysis of Social Bot Partisan Behaviorcited by 9
  11. 2016Ideology Detection for Twitter Users with Heterogeneous Types of Linkscited by 4
  12. 2020Application of Pre-training Models in Named Entity Recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 22; 2025: 9; 2026: 2.

34 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024Best Practices for Text Annotation with Large Language Modelscited by 34
  3. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  4. 2024Web Retrieval Agents for Evidence-Based Misinformation Detectioncited by 13
  5. 2024GPTs Are Multilingual Annotators for Sequence Generation Taskscited by 11
  6. 2024Automated Text Scoring in the Age of Generative AI for the GPU-poorcited by 10
  7. 2024A Guide to Misinformation Detection Data and Evaluationcited by 10
  8. 2023Are Chatbots Reliable Text Annotators? Sometimescited by 9
  9. 2024Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigationcited by 9
  10. 2025ScholaWrite: A Dataset of End-to-End Scholarly Writing Processcited by 8
  11. 2024Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotatorscited by 7
  12. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
22 more
  1. 2025Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detectioncited by 5
  2. 2024APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPTcited by 4
  3. 2024Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Taskscited by 4
  4. 2024Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Modelscited by 4
  5. 2024LlamBERT: Large-scale low-cost data annotation in NLPcited by 3
  6. 2025Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Reviewcited by 3
  7. 2024Leveraging Open-Source Large Language Models for Native Language Identificationcited by 2
  8. 2024Data-Centric AI Governance: Addressing the Limitations of Model-Focused Policiescited by 2
  9. 2024Your Extreme Multi-label Classifier is Secretly a Hierarchical Text Classifier for Freecited by 2
  10. 2024ScamSpot: Fighting Financial Fraud in Instagram Commentscited by 1
  11. 2024Illuminating Blind Spots of Language Models with Targeted Agent-in-the-Loop Synthetic Datacited by 1
  12. 2024Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneckcited by 1
  13. 2025Rethinking the Understanding Ability across LLMs through Mutual Informationcited by 1
  14. 2025Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Modelscited by 1
  15. 2026What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systemscited by 1
  16. 2024Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Datacited by 0
  17. 2024Suicide Risk Assessment on Social Media with Semi-Supervised Learningcited by 0
  18. 2025A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classificationcited by 0
  19. 2025Large Language Models in the Task of Automatic Validation of Text Classifier Predictionscited by 0
  20. 2025A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goalscited by 0
  21. 2025Measuring Negative Campaigning across Languages with Large Language Models: A Study of 18 Million Tweets in 19 Countriescited by 0
  22. 2026Selecting Language Models for Social Science: Start Small, Start Open, and Validatecited by 0