# Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science

arXiv: https://arxiv.org/abs/2305.14310 · HTML: https://mlsanity.com/paper/2305.14310/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Detection and Resolution of Rumours in Social Media: A Survey](https://mlsanity.com/paper/1704.00656/) (2017)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours](https://mlsanity.com/paper/1704.05972/) (2017)
- [Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks](https://mlsanity.com/paper/2305.05862/) (2023)
- [Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark](https://mlsanity.com/paper/2304.11085/) (2023)
- [Evaluation of ChatGPT for NLP-based Mental Health Applications](https://mlsanity.com/paper/2303.15727/) (2023)
- [Toxicity Detection with Generative Prompt-based Inference](https://mlsanity.com/paper/2205.12390/) (2022)
- [Automatically Identifying Complaints in Social Media](https://mlsanity.com/paper/1906.03890/) (2019)
- [Winds of Change: Impact of COVID-19 on Vaccine-related Opinions of Twitter users](https://mlsanity.com/paper/2111.10667/) (2021)
- [VaxxHesitancy: A Dataset for Studying Hesitancy towards COVID-19 Vaccination on Twitter](https://mlsanity.com/paper/2301.06660/) (2023)
- [Automatic Identification and Classification of Bragging in Social Media](https://mlsanity.com/paper/2203.05840/) (2022)

## Built on by (most cited first, 19 of 19)

- [Benchmarking Prompt Sensitivity in Large Language Models](https://mlsanity.com/paper/2502.06065/) (2025)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [On Sarcasm Detection with OpenAI GPT-based Models](https://mlsanity.com/paper/2312.04642/) (2023)
- [Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science](https://mlsanity.com/paper/2409.14673/) (2024)
- [Generative AI and US Intellectual Property Law](https://mlsanity.com/paper/2311.16023/) (2023)
- [Language Evolution for Evading Social Media Regulation via LLM-based Multi-agent Simulation](https://mlsanity.com/paper/2405.02858/) (2024)
- [ChatGPT Alternative Solutions: Large Language Models Survey](https://mlsanity.com/paper/2403.14469/) (2024)
- [Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detection](https://mlsanity.com/paper/2407.12879/) (2024)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [Generative AI in Sociological Research: State of the Discipline](https://mlsanity.com/paper/2511.16884/) (2025)
- [A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processing](https://mlsanity.com/paper/2403.02504/) (2024)
- [StreamProfileBench: A Benchmark for Fine-Grained User Profile Inference in Real-World Streaming Scenarios](https://mlsanity.com/paper/2605.25758/) (2026)
- [How to Ask the AI: A User Perspective Survey for Large Language Model Prompting](https://mlsanity.com/paper/2608.07494/) (2026)
- [Assessing how hyperparameters impact Large Language Models' sarcasm detection performance](https://mlsanity.com/paper/2504.06166/) (2025)
- [Exploring the Potential of Large Language Models in Fine-Grained Review Comment Classification](https://mlsanity.com/paper/2508.09832/) (2025)
- [From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models](https://mlsanity.com/paper/2509.09303/) (2025)
- [What Helps Language Models Predict Human Beliefs: Demographics or Prior Stances?](https://mlsanity.com/paper/2511.18616/) (2025)
- [Soft Contextualized Encoder For User Defined Text Classification](https://mlsanity.com/paper/2601.03450/) (2026)
- [Observational Equivalence of LLM and Human Annotation](https://mlsanity.com/paper/2609.22133/) (2026)

## Abstract (arXiv)

Instruction-tuned Large Language Models (LLMs) have exhibited impressive language understanding and the capacity to generate responses that follow specific prompts. However, due to the computational demands associated with training these models, their applications often adopt a zero-shot setting. In this paper, we evaluate the zero-shot performance of two publicly accessible LLMs, ChatGPT and OpenAssistant, in the context of six Computational Social Science classification tasks, while also investigating the effects of various prompting strategies. Our experiments investigate the impact of prompt complexity, including the effect of incorporating label definitions into the prompt; use of synonyms for label names; and the influence of integrating past memories during foundation model training. The findings indicate that in a zero-shot setting, current LLMs are unable to match the performance of smaller, fine-tuned baseline transformer models (such as BERT-large). Additionally, we find that different prompting strategies can significantly affect classification accuracy, with variations in accuracy and F1 scores exceeding 10%.