mlsanityData to 14 Oct 2026

LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability

arXiv 2606.15610 ↗First posted 14 Jun 2026cs.CL astro-ph.IM cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 10 of its 27 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

10 of 27 references.

References · 10

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2018Datasheets for Datasetscited by 1,386
  4. 2018Model Cards for Model Reportingcited by 1,080
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  8. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
  9. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 10
  10. 2026LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panelscited by 1
  3. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  4. 2026Three Ways Classical Test Theory Can Mislead About LLM Judgescited by 0
  5. 2026REMEDY: How Far Is Video Generation from Medical Education World Models?cited by 0
  6. 2026The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generatingcited by 0