mlsanityData to 7 Oct 2026

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

arXiv 2607.08535 ↗First posted 9 Jul 2026cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 11 of its 22 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

11 of 22 references.

References · 11

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  6. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  7. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  10. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 24
  11. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panelscited by 1
  2. 2026Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgetscited by 0
  3. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidencecited by 0
  4. 2026Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviewscited by 0
  5. 2026Ask Which, Not How Good: Sizing Benchmarks Scored by an LLMcited by 0
  6. 2026When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Driftcited by 0