# Empirical Prompt Engineering for Construct Identification with Large Language Models

arXiv: https://arxiv.org/abs/2512.03818 · HTML: https://mlsanity.com/paper/2512.03818/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 76 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [Evaluating GPT-3 Generated Explanations for Hateful Content Moderation](https://mlsanity.com/paper/2305.17680/) (2023)
- [Large language models can segment narrative events similarly to humans](https://mlsanity.com/paper/2301.10297/) (2023)
- [Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models](https://mlsanity.com/paper/2401.00284/) (2023)
- [The Unreasonable Effectiveness of Eccentric Automatic Prompts](https://mlsanity.com/paper/2402.10949/) (2024)
- [Grading Massive Open Online Courses Using Large Language Models](https://mlsanity.com/paper/2406.11102/) (2024)

## Abstract (arXiv)

Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong performance on text classification tasks. However, LLM classifications are highly responsive to prompt wording, particularly, as we show, in domains like psychology, where constructs are often latent, complex, and theory driven. Here, we present and evaluate a systematic framework for improving psychological construct identification through prompt engineering. We combinatorially generate prompts by appending random selections of multiple variants of construct definitions, task instructions, coding guidance, and examples. Empirically selecting the highest performing of these combinations in a training dataset substantially improves alignment between LLM and human classifications. In contrast, prompting techniques such as personas, chain-of-thought reasoning, and explanations provide smaller and less consistent improvements. This finding holds across multiple models and constructs. Overall, the approach we describe offers a practical, systematic, and theory-aware method for increasing the alignment between human and LLM classifications in settings where validity is critical.