mlsanityData to 7 Oct 2026

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels

arXiv 2609.17857 ↗First posted 15 Sep 2026cs.CL cs.AI cs.CYcited by 1

Not on the map. Why?

Abstract (arXiv)

Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confounded with candidate quality and correlates with Bradley-Terry ability at r = 0.95. We derive a corrected estimator that holds the candidate family fixed and compares judges. All four families then show a positive same-family lift (3.4-8.4 percentage points), with global FPS 0.067 (95% CI [0.053, 0.084], permutation p = 0.0002). The effect remains under panel-based quality controls, an independent human-consensus anchor, and a float16 judging replication. Judge-side likelihood is closely related to the effect: adding likelihood advantage reduces the controlled coefficient by 61%, which we treat as descriptive attenuation rather than causal mediation. Position is a separate failure mode. Across the panel, 55.4% of AB/BA pairs reverse, and reversal above 50% is incompatible with a simple independent content-noise model. Relative to a family-balanced reference, panel composition changes 18.5% of pairwise outcomes. A complete reproducibility archive has been prepared for public release.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

22 of 32 references.

References · 22

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  7. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  8. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  9. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  10. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  11. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  12. 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
10 more
  1. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  2. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  3. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
  4. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
  5. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 9
  6. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 7
  7. 2026LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluationcited by 6
  8. 2026When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliabilitycited by 6
  9. 2026QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofscited by 3
  10. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of "AI-AI Bias" Show No Detectable Own-Model Premiumcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
  2. 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
  3. 2026How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  5. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  6. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  7. 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
  8. 2026The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment