Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Not on the map. Why?
Abstract (arXiv)
We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 9 of its 11 references that are in the map. First posted on arXiv 5 Mar 2026.
Builds on
9 of 11 references.
References · 9
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
- 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
- 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
- 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
- 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 9
- 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
- 2026A Finite-Calibration Regime Map for LLM Judge Panelscited by 2
- 2026LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categoriescited by 1
- 2026When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Textcited by 0
- 2026MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluatorscited by 0
- 2026Unmentioned Checklist Findings Change How Reinforcement Learning Appears to Improve Chest Radiograph Report Checkingcited by 0
- 2026Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditionscited by 0
- 2026Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions6 Oct 2026
- 2026Unmentioned Checklist Findings Change How Reinforcement Learning Appears to Improve Chest Radiograph Report Checking4 Oct 2026
- 2026MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators18 Sep 2026
- 2026When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text10 Sep 2026
- 2026A Finite-Calibration Regime Map for LLM Judge Panels31 May 2026
- 2026LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories29 May 2026
- 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges7 May 2026
- 2026When LLM Judge Scores Look Good but Best-of-N Decisions Fail12 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026All Verdicts are Not Equal: Rethinking LLM Judge Reliability
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
- 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
- 2026Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence
- 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
- 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
- 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts
- 2026BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories