# LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability

arXiv: https://arxiv.org/abs/2606.15610 · HTML: https://mlsanity.com/paper/2606.15610/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory](https://mlsanity.com/paper/2602.00521/) (2026)
- [Quantifying and Mitigating Self-Preference Bias of LLM Judges](https://mlsanity.com/paper/2604.22891/) (2026)
- [LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy](https://mlsanity.com/paper/2603.14893/) (2026)

## Built on by (most cited first, 6 of 6)

- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)
- [Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels](https://mlsanity.com/paper/2609.17857/) (2026)
- [A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality](https://mlsanity.com/paper/2609.18005/) (2026)
- [Three Ways Classical Test Theory Can Mislead About LLM Judges](https://mlsanity.com/paper/2609.29709/) (2026)
- [REMEDY: How Far Is Video Generation from Medical Education World Models?](https://mlsanity.com/paper/2609.32460/) (2026)
- [The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating](https://mlsanity.com/paper/2610.00054/) (2026)

## Abstract (arXiv)

Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.