# Best Practices for Text Annotation with Large Language Models

arXiv: https://arxiv.org/abs/2402.05129 · HTML: https://mlsanity.com/paper/2402.05129/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models](https://mlsanity.com/paper/2306.04757/) (2023)
- [Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators](https://mlsanity.com/paper/2307.05532/) (2023)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models](https://mlsanity.com/paper/2401.00284/) (2023)
- [How to use LLMs for Text Analysis](https://mlsanity.com/paper/2307.13106/) (2023)

## Built on by (most cited first, 30 of 36)

- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?](https://mlsanity.com/paper/2506.19467/) (2025)
- [Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin](https://mlsanity.com/paper/2410.09245/) (2024)
- [Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses](https://mlsanity.com/paper/2504.02080/) (2025)
- [A Universal Prompting Strategy for Extracting Process Model Information from Natural Language Text using Large Language Models](https://mlsanity.com/paper/2407.18540/) (2024)
- [Position: LLMs Can be Good Tutors in English Education](https://mlsanity.com/paper/2502.05467/) (2025)
- [Can Large Language Models (or Humans) Disentangle Text?](https://mlsanity.com/paper/2403.16584/) (2024)
- [Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification](https://mlsanity.com/paper/2507.05010/) (2025)
- [Addressing Longstanding Challenges in Cognitive Science with Language Models](https://mlsanity.com/paper/2511.00206/) (2025)
- [Detecting Calls to Action in Multimodal Content: Analysis of the 2021 German Federal Election Campaign on Instagram](https://mlsanity.com/paper/2409.02690/) (2024)
- [Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciences](https://mlsanity.com/paper/2507.21831/) (2025)
- [Generative Artificial Intelligence in Qualitative Research Methods: Between Hype and Risks?](https://mlsanity.com/paper/2511.08461/) (2025)
- [Scaling Technology Acceptance Analysis with Large Language Model (LLM) Annotation Systems](https://mlsanity.com/paper/2407.00702/) (2024)
- [Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences](https://mlsanity.com/paper/2504.03822/) (2025)
- [Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications](https://mlsanity.com/paper/2505.14918/) (2025)
- [Measuring Negative Campaigning across Languages with Large Language Models: A Study of 18 Million Tweets in 19 Countries](https://mlsanity.com/paper/2507.17636/) (2025)
- [Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation](https://mlsanity.com/paper/2603.13891/) (2026)
- [Beyond the Link: Assessing LLMs' ability to Classify Political Content across Global Media](https://mlsanity.com/paper/2506.17435/) (2025)
- [Argument Quality Annotation and Gender Bias Detection in Financial Communication through Large Language Models](https://mlsanity.com/paper/2508.08262/) (2025)
- [Generative Large Language Models (gLLMs) in Content Analysis: A Practical Guide for Communication Research](https://mlsanity.com/paper/2510.24337/) (2025)
- [Do Androids Dream of Unseen Puppeteers? Probing for a Conspiracy Tendencies in Large Language Models](https://mlsanity.com/paper/2511.03699/) (2025)
- [Applying Large Language Models to Characterize Public Narratives](https://mlsanity.com/paper/2511.13505/) (2025)
- [Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety](https://mlsanity.com/paper/2512.06227/) (2025)
- [Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique](https://mlsanity.com/paper/2601.09905/) (2026)
- [A Human-Centered Workflow for Using Large Language Models in Content Analysis](https://mlsanity.com/paper/2603.19271/) (2026)
- [Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation](https://mlsanity.com/paper/2603.26898/) (2026)
- [Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study](https://mlsanity.com/paper/2604.01615/) (2026)
- [Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction](https://mlsanity.com/paper/2604.10786/) (2026)
- [Mapping Election Toxicity on Social Media across Issue, Ideology, and Psychosocial Dimensions](https://mlsanity.com/paper/2604.16765/) (2026)
- [Seeing Candidates at Scale: Multimodal LLMs for Visual Political Communication on Instagram](https://mlsanity.com/paper/2604.19489/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have ushered in a new era of text annotation, as their ease-of-use, high accuracy, and relatively low costs have meant that their use has exploded in recent months. However, the rapid growth of the field has meant that LLM-based annotation has become something of an academic Wild West: the lack of established practices and standards has led to concerns about the quality and validity of research. Researchers have warned that the ostensible simplicity of LLMs can be misleading, as they are prone to bias, misunderstandings, and unreliable results. Recognizing the transformative potential of LLMs, this paper proposes a comprehensive set of standards and best practices for their reliable, reproducible, and ethical use. These guidelines span critical areas such as model selection, prompt engineering, structured prompting, prompt stability analysis, rigorous model validation, and the consideration of ethical and legal implications. The paper emphasizes the need for a structured, directed, and formalized approach to using LLMs, aiming to ensure the integrity and robustness of text annotation practices, and advocates for a nuanced and critical engagement with LLMs in social scientific research.