mlsanityData to 13 Oct 2026

Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research

arXiv 2503.22040 ↗First posted 27 Mar 2025cs.CL cs.CYcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have the potential to revolutionize computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks associated with using LLMs for text classification tasks, using social movement studies as an example. We propose a framework for social scientists to incorporate LLMs into text annotation, either as the primary coding decision-maker or as a coding assistant. This framework offers researchers tools to develop the potential best-performing prompt, and to systematically examine and report the validity and reliability of LLMs as a methodological tool. Additionally, we evaluate and discuss its epistemic risks associated with validity, reliability, replicability, and transparency. We conclude with several practical guidelines for using LLMs in text annotation tasks and offer recommendations for more effectively communicating epistemic risks in research.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 24 of its 64 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

24 of 64 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  7. 2020Longformer: The Long-Document Transformercited by 3,393
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,181
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 857
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 800
  11. 2023How is ChatGPT's behavior changing over time?cited by 268
  12. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
12 more
  1. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  2. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  3. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 221
  4. 2023Large Language Model Guided Tree-of-Thoughtcited by 219
  5. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  6. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  7. 2018The Geometry of Culture: Analyzing Meaning through Word Embeddingscited by 87
  8. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
  9. 2024Prompt Stability Scoring for Text Annotation with Large Language Modelscited by 14
  10. 2023Improving Large Language Models for Clinical Named Entity Recognition via Prompt Engineeringcited by 7
  11. 2021Introduction to Neural Transfer Learning with Transformers for Social Science Text Analysiscited by 6
  12. 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?cited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Vibe Researching as Wolf Coming: Can AI Agents with Skills Replace or Augment Social Scientists?cited by 4
  2. 2025Generative AI in Sociological Research: State of the Disciplinecited by 2
  3. 2025Large Language Model-Informed Feature Discovery Improves Prediction and Interpretation of Credibility Perceptions of Visual Contentcited by 0
  4. 2026Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudicationcited by 0