# Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts

arXiv: https://arxiv.org/abs/2407.10747 · HTML: https://mlsanity.com/paper/2407.10747/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels](https://mlsanity.com/paper/2406.17633/) (2024)
- [Stance Detection: A Practical Guide to Classifying Political Beliefs in Text](https://mlsanity.com/paper/2305.01723/) (2023)
- [Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence](https://mlsanity.com/paper/2105.12936/) (2021)
- [Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement](https://mlsanity.com/paper/2310.18440/) (2023)
- [Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways](https://mlsanity.com/paper/2406.11980/) (2024)
- [Synthetically generated text for supervised text analysis](https://mlsanity.com/paper/2303.16028/) (2023)
- [Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers](https://mlsanity.com/paper/2310.18360/) (2023)

## Built on by (most cited first, 30 of 30)

- ['Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants](https://mlsanity.com/paper/2409.19430/) (2024)
- [From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning](https://mlsanity.com/paper/2507.22028/) (2025)
- [Value Alignment of Social Media Ranking Algorithms](https://mlsanity.com/paper/2509.14434/) (2025)
- [Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification](https://mlsanity.com/paper/2507.05010/) (2025)
- [Feeding LLM Annotations to BERT Classifiers at Your Own Risk](https://mlsanity.com/paper/2504.15432/) (2025)
- [What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification](https://mlsanity.com/paper/2510.03541/) (2025)
- [Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer](https://mlsanity.com/paper/2606.11018/) (2026)
- [Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs](https://mlsanity.com/paper/2606.28574/) (2026)
- [Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning](https://mlsanity.com/paper/2509.07768/) (2025)
- [Op-Fed: Opinion, Stance, and Monetary Policy Annotations on FOMC Transcripts Using Active Learning](https://mlsanity.com/paper/2509.13539/) (2025)
- [How Persuasive is Your Context?](https://mlsanity.com/paper/2509.17879/) (2025)
- [LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text](https://mlsanity.com/paper/2604.14321/) (2026)
- [When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding](https://mlsanity.com/paper/2606.06781/) (2026)
- [Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements](https://mlsanity.com/paper/2608.18294/) (2026)
- [When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability](https://mlsanity.com/paper/2609.07987/) (2026)
- [Forecasting Commencing Enrolments Under Data Sparsity: A Zero-Shot Time Series Foundation Models Framework for Higher Education Planning](https://mlsanity.com/paper/2602.12120/) (2026)
- [Agentic Framework for Political Biography Extraction](https://mlsanity.com/paper/2603.18010/) (2026)
- [A conceptual framework for ideology in online discourse beyond the left and right](https://mlsanity.com/paper/2603.18945/) (2026)
- [Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation](https://mlsanity.com/paper/2603.26898/) (2026)
- [Multicalibration for Unbiased Model-Based Prevalence Estimation](https://mlsanity.com/paper/2604.21549/) (2026)
- [LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics](https://mlsanity.com/paper/2605.13412/) (2026)
- [Toward Responsible and Epistemically Grounded Multilingual LLMs for Computational Social Science and Humanities](https://mlsanity.com/paper/2606.00596/) (2026)
- [Validating LLMs in social science: Epistemic threats and emerging norms](https://mlsanity.com/paper/2607.07915/) (2026)
- [Dimensionality in Satisfaction Ratings](https://mlsanity.com/paper/2607.11026/) (2026)
- [CALICO: A Human-Centered, Codebook-Aligned System for Annotation](https://mlsanity.com/paper/2609.14726/) (2026)
- [A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance](https://mlsanity.com/paper/2609.16592/) (2026)
- [Making Political Text Scaling Comparable: Infrastructure and Hyperparameter Sensitivity for 17 Algorithms](https://mlsanity.com/paper/2609.17602/) (2026)
- [Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation](https://mlsanity.com/paper/2609.26926/) (2026)
- [EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues](https://mlsanity.com/paper/2609.27043/) (2026)
- [Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment](https://mlsanity.com/paper/2610.16757/) (2026)

## Abstract (arXiv)

Codebooks -- documents that operationalize concepts and outline annotation procedures -- are used almost universally by social scientists when coding political texts. To code these texts automatically, researchers are increasing turning to generative large language models (LLMs). However, there is limited empirical evidence on whether "off-the-shelf" LLMs faithfully follow real-world codebook operationalizations and measure complex political constructs with sufficient accuracy. To address this, we gather and curate three real-world political science codebooks -- covering protest events, political violence and manifestos -- along with their unstructured texts and human labels. We also propose a five-stage framework for codebook-LLM measurement: preparing a codebook for both humans and LLMs, testing LLMs' basic capabilities on a codebook, evaluating zero-shot measurement accuracy (i.e. off-the-shelf performance), analyzing errors, and further (parameter-efficient) supervised training of LLMs. We provide an empirical demonstration of this framework using our three codebook datasets and several pretrained 7-12 billion open-weight LLMs. We find current open-weight LLMs have limitations in following codebooks zero-shot, but that supervised instruction tuning can substantially improve performance. Rather than suggesting the "best" LLM, our contribution lies in our codebook datasets, evaluation framework, and guidance for applied researchers who wish to implement their own codebook-LLM measurement projects.