Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciences
Not on the map. Why?
Abstract (arXiv)
LLMs are seeing widespread use for task automation, including automated coding in the social sciences. However, even though researchers have proposed different prompting strategies, their effectiveness varies across LLMs and tasks. Often trial and error practices are still widespread. Our study aims to fill this gap and evaluate how LLMs can be used in a systematic and transparent way to produce reliable codings in content analyses. We propose HALC-a general pipeline that allows for the systematic and reliable construction of prompts for any given coding task and model. We develop this pipeline based on current literature and findings of a prestudy investigating consistency and influencing factors of LLM codings. We also apply HALC on two other datasets covering different thematic contexts, document types, languages, and coding units to test its applicability. Based on more than three million LLM requests, our results demonstrate that the pipeline is capable of identifying prompts for reliable codings in different settings. We also discuss shortcomings and further potential for development.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 63 references that are in the map. First posted on arXiv 29 Jul 2025.
Builds on
24 of 63 references.
References · 24
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
- 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
- 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
- 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
12 more
- 2025Chain of Draft: Thinking Faster by Writing Lesscited by 297
- 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 97
- 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 93
- 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
- 2023Automated Annotation with Generative AI Requires Validationcited by 76
- 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
- 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
- 2024Best Practices for Text Annotation with Large Language Modelscited by 36
- 2021A Systematic Review of Reproducibility Research in Natural Language Processingcited by 34
- 2023Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuningcited by 32
- 2025Learning to Generate Structured Output with Schema Reinforcement Learningcited by 29
- 2023Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Modelscited by 12
Built on by
Citing papers per year: 2026: 2.
2 papers.