mlsanityData to 14 Oct 2026

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

arXiv 2608.24419 ↗First posted 25 Aug 2026cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-changing intervention types and 5 register-only controls, with intervention direction determined by human annotators and generation, verification, and judging assigned to disjoint model families. At matched invariance S >= 0.90, judges average S = 0.945 but R = 0.319. Sensitivity also differs between scope and strength edits: R_scope = 0.383 versus R_strength = 0.262, a +0.121 gap with the same sign for all 7 judges. We further audit five public label sets and find that surface-only predictors reproduce 55%-67% of labels in paired mode, including 67.4% of MT-Bench human votes. These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated, motivating joint reporting of invariance and sensitivity and auditing the validation set itself.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

37 of 57 references.

References · 37

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  8. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  12. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
25 more
  1. 2019Measurement and Fairnesscited by 308
  2. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  3. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  4. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  6. 2022Defining and Characterizing Reward Hackingcited by 203
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  8. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  9. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  10. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  11. 2025Generalization Bias in Large Language Model Summarization of Scientific Researchcited by 30
  12. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  13. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  14. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 24
  15. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  16. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  17. 2023A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documentscited by 16
  18. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 15
  19. 2025CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?cited by 12
  20. 2025Rectifying Shortcut Behaviors in Preference-based Reward Learningcited by 6
  21. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  22. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 4
  23. 2026RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluationcited by 3
  24. 2026Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Biascited by 3
  25. 2026The strength of clinical evidence is recoverable from language model representations but not from their stated gradescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Auditing Proxy-Based Validation Across Text Spanscited by 0
  2. 2026Audience-Bound Persistent Memory: Authorization Across the Memory Lifecyclecited by 0
  3. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0
  4. 2026Unmentioned Checklist Findings Change How Reinforcement Learning Appears to Improve Chest Radiograph Report Checkingcited by 0