# Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Research

arXiv: https://arxiv.org/abs/2411.00890 · HTML: https://mlsanity.com/paper/2411.00890/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 28 of its 66 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification](https://mlsanity.com/paper/2406.08660/) (2024)
- [Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking](https://mlsanity.com/paper/2311.06102/) (2023)
- [Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations](https://mlsanity.com/paper/2404.10779/) (2024)
- [CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics](https://mlsanity.com/paper/2406.15477/) (2024)
- [Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels](https://mlsanity.com/paper/2406.17633/) (2024)
- [FLM-101B: An Open LLM and How to Train It with 100K Budget](https://mlsanity.com/paper/2309.03852/) (2023)
- [Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production](https://mlsanity.com/paper/2312.14972/) (2023)
- [Closing the gap between open-source and commercial large language models for medical evidence summarization](https://mlsanity.com/paper/2408.00588/) (2024)
- [Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation](https://mlsanity.com/paper/2406.12114/) (2024)
- [Prompt Stability Scoring for Text Annotation with Large Language Models](https://mlsanity.com/paper/2407.02039/) (2024)
- [From Words to Molecules: A Survey of Large Language Models in Chemistry](https://mlsanity.com/paper/2402.01439/) (2024)
- [Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data](https://mlsanity.com/paper/2403.19031/) (2024)
- [Enhancing LLM-Based Text Classification in Political Science: Automatic Prompt Optimization and Dynamic Exemplar Selection for Few-Shot Learning](https://mlsanity.com/paper/2409.01466/) (2024)
- [SUDO: a framework for evaluating clinical artificial intelligence systems without ground-truth annotations](https://mlsanity.com/paper/2403.17011/) (2024)
- [Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification](https://mlsanity.com/paper/2404.10757/) (2024)
- [FAIR GPT: A virtual consultant for research data management in ChatGPT](https://mlsanity.com/paper/2410.07108/) (2024)

## Built on by (most cited first, 6 of 6)

- [Deep literature reviews: an application of fine-tuned language models to migration research](https://mlsanity.com/paper/2504.13685/) (2025)
- [The Human Flourishing Geographic Index: A County-Level Dataset for the United States, 2013--2023](https://mlsanity.com/paper/2511.03915/) (2025)
- [Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions](https://mlsanity.com/paper/2511.03047/) (2025)
- [Two Americas of Well-Being: Divergent Rural-Urban Patterns of Life Satisfaction and Happiness from 2.6 B Social Media Posts](https://mlsanity.com/paper/2511.10542/) (2025)
- [Spatial Heterogeneity in Climate Risk and Human Flourishing: An Exploration with Generative AI](https://mlsanity.com/paper/2601.20880/) (2026)
- [Geospatial Metadata Improves Discoverability by Connecting Datasets Across Scientific Disciplines](https://mlsanity.com/paper/2609.16498/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are distinguished by their architecture, which dictates their parameter size and performance capabilities. Social scientists have increasingly adopted LLMs for text classification tasks, which are difficult to scale with human coders. While very large, closed-source models often deliver superior performance, their use presents significant risks. These include lack of transparency, potential exposure of sensitive data, challenges to replicability, and dependence on proprietary systems. Additionally, their high costs make them impractical for large-scale research projects. In contrast, open-source models, although available in various sizes, may underperform compared to commercial alternatives if used without further fine-tuning. However, open-source models offer distinct advantages: they can be run locally (ensuring data privacy), fine-tuned for specific tasks, shared within the research community, and integrated into reproducible workflows. This study demonstrates that small, fine-tuned open-source LLMs can achieve equal or superior performance to models such as ChatGPT-4. We further explore the relationship between training set size and fine-tuning efficacy in open-source models. Finally, we propose a hybrid workflow that leverages the strengths of both open and closed models, offering a balanced approach to performance, transparency, and reproducibility.