mlsanityData to 14 Oct 2026

Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

arXiv 2606.28574 ↗First posted 26 Jun 2026cs.CL cs.AI cs.CYcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

When a large language model (LLM) codes a construct in text as a human annotator would, that agreement makes the LLM a reliable coder. Yet reliability leaves construct validity untouched. The instrument may be theory-naive, reaching the code through a correlate that meets none of the demands the construct's theory makes, and no current method tells that apart from genuine measurement. We propose grain calibration as a method that closes the gap. It decomposes a construct into clause-level components, tests each against the text with extractive evidence, and combines the results through an explicit, theory-derived rule. Because the rule is stated rather than lodged in one opaque pass, its structure is evidence about the process rather than the output. It shows which components settled a code, and, when the code is wrong, whether a component was missed or an adjacent construct mistaken for it. Validation shifts from scoring an instrument's outputs against an annotator to showing that the instrument runs on the construct its theory specifies.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 58 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

26 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks26 Jan
  2. 2023Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning1 Jun
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb

Other references · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  5. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  6. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 119
  7. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  8. 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
  9. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  10. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 39
  11. 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 30
  12. 2023Using Large Language Models for Qualitative Analysis can Introduce Serious Biascited by 22
11 more
  1. 2024Prompt Stability Scoring for Text Annotation with Large Language Modelscited by 15
  2. 2025Do AI Models Perform Human-like Abstract Reasoning Across Modalities?cited by 10
  3. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  4. 2025A validity-guided workflow for robust large language model research in psychologycited by 8
  5. 2026Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networkscited by 6
  6. 2024Show Less, Instruct More: Enriching Prompts with Definitions and Guidelines for Zero-Shot NERcited by 5
  7. 2025Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitionscited by 4
  8. 2023ValiText -- a unified validation framework for computational text-based measures of social constructscited by 2
  9. 2025Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Studycited by 2
  10. 2026Enhancing LLM-Based Data Annotation with Error Decompositioncited by 1
  11. 2026Measuring What Cannot Be Surveyed: LLMs as Instruments for Latent Cognitive Variables in Labor Economicscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIAcited by 0
  2. 2026Auditing Proxy-Based Validation Across Text Spanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  2. 2026Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence
  3. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  4. 2026Rethinking Uncertainty Evaluation in Large Language Models
  5. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
  6. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  7. 2025CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
  8. 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs