mlsanityData to 7 Oct 2026

Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models

arXiv 2407.10645 ↗First posted 15 Jul 2024cs.CL cs.AI cs.CYcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models have recently been applied to text annotation tasks from social sciences, equalling or surpassing the performance of human workers at a fraction of the cost. However, no inquiry has yet been made on the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/ .

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 26 of its 54 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

26 of 54 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  7. 2023Large Language Models as Optimizerscited by 830
  8. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  9. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  10. 2022Can language models learn from explanations in context?cited by 248
  11. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  12. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
14 more
  1. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  2. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  3. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  4. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  5. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  6. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  7. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 97
  8. 2020We Can Detect Your Bias: Predicting the Political Ideology of News Articlescited by 92
  9. 2020A Survey on Text Classification: From Shallow to Deep Learningcited by 80
  10. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  11. 2024A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregationcited by 23
  12. 2024RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agentscited by 15
  13. 2024The Unreasonable Effectiveness of Eccentric Automatic Promptscited by 12
  14. 2023Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Modelscited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Says Who? Effective Zero-Shot Annotation of Focalizationcited by 2
  2. 2025A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languagescited by 2