# Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks

arXiv: https://arxiv.org/abs/2408.01346 · HTML: https://mlsanity.com/paper/2408.01346/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 2 Aug 2024.

## Builds on (most cited first, 7 of 7)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)

## Built on by (most cited first, 6 of 6)

- [Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale](https://mlsanity.com/paper/2505.23785/) (2025)
- [Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science](https://mlsanity.com/paper/2409.14673/) (2024)
- [A Picture is Worth a Thousand Prompts? Efficacy of Iterative Human-Driven Prompt Refinement in Image Regeneration Tasks](https://mlsanity.com/paper/2504.20340/) (2025)
- [MemoCoder: Automated Function Synthesis using LLM-Supported Agents](https://mlsanity.com/paper/2507.18812/) (2025)
- [Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content](https://mlsanity.com/paper/2602.19177/) (2026)
- [CLPIPS: A Personalized Metric for AI-Generated Image Similarity](https://mlsanity.com/paper/2604.01234/) (2026)

## Abstract (arXiv)

Large Language Models are expressive tools that enable complex tasks of text understanding within Computational Social Science. Their versatility, while beneficial, poses a barrier for establishing standardized best practices within the field. To bring clarity on the values of different strategies, we present an overview of the performance of modern LLM-based classification methods on a benchmark of 23 social knowledge tasks. Our results point to three best practices: select models with larger vocabulary and pre-training corpora; avoid simple zero-shot in favor of AI-enhanced prompting; fine-tune on task-specific data, and consider more complex forms instruction-tuning on multiple datasets only when only training data is more abundant.