mlsanityData to 7 Oct 2026

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

arXiv 2602.00521 ↗First posted 31 Jan 2026cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 52 of its 77 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

52 of 77 references.

References · 52

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2024A Survey on LLM-as-a-Judgecited by 1,384
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
38 more
  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  5. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  7. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  8. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  9. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  10. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  11. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  12. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  13. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
  14. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  15. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  16. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  17. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  18. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  19. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  20. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 75
  21. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  22. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  23. 2024Self-Taught Evaluatorscited by 70
  24. 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 54
  25. 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 52
  26. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  27. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  28. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  29. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  30. 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 23
  31. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  32. 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18
  33. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  34. 2025On the Effectiveness of LLM-as-a-judge for Code Generation and Summarizationcited by 17
  35. 2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 16
  36. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  37. 2025Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?cited by 9
  38. 2025Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?cited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8
  2. 2026LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluationcited by 6
  3. 2026Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?cited by 3
  4. 2026RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficultycited by 0
  5. 2026Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theorycited by 0
  6. 2026JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errorscited by 0
  7. 2026LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distributioncited by 0
  8. 2026NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervisioncited by 0
  9. 2026Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistencecited by 0
  10. 2026Rating the Raters: Rasch Measurement Theory for LLM Evaluationcited by 0
  11. 2026Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpointscited by 0
  12. 2026Ask Which, Not How Good: Sizing Benchmarks Scored by an LLMcited by 0
1 more
  1. 2026Three Ways Classical Test Theory Can Mislead About LLM Judgescited by 0