mlsanityData to 7 Oct 2026

A Finite-Calibration Regime Map for LLM Judge Panels

arXiv 2606.01034 ↗First posted 31 May 2026cs.CL stat.MEcited by 2

Not on the map. Why?

Abstract (arXiv)

Deploying an LLM judge panel spends human labels on fitting a calibrator, constructing candidate judge paths, and validating which candidate to deploy. We study when finite labels should support a low-dimensional stacker or reliability model, and when an unrestricted joint output table is worth its cell-count and unseen-pattern cost. We cast this as a finite-calibration regime map and instantiate it as Finite-Calibration Panel Selection (FCPS), a validation selector over judge path, deployed panel size, and aggregator family with support diagnostics. Across RewardBench, LLMBar, SummEval, and Arena100K with a seven-judge pool, scalar/reliability aggregation has lower MSE than unrestricted joint-table calibration in 16 of 20 real dataset--budget cells by point estimate, while paired 95% intervals exclude zero in 11 cells; richer backoff/shrinkage tables narrow some gaps while preserving the finite-support bottleneck. Controlled calibration-growth data show the opposite regime: when labels contain a six-way interaction, the selected table grows to the interaction-bearing prefix and its MSE falls from 0.224 to 0.061 once unseen mass vanishes. The practical deployment question is whether the next judge's information is estimable under the available human labels.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

33 of 45 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
21 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  2. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  3. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  4. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  5. 2025Atla Selene Mini: A General Purpose Evaluation Modelcited by 27
  6. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  7. 2023Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMscited by 23
  8. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  9. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
  10. 2025Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluationscited by 16
  11. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 12
  12. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  13. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  14. 2026Judge Reliability Harness: Stress Testing the Reliability of LLM Judgescited by 6
  15. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 4
  16. 2026Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluationcited by 3
  17. 2026On Cost-Effective LLM-as-a-Judge Improvement Techniquescited by 3
  18. 2026Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidencecited by 3
  19. 2025Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judgecited by 2
  20. 2025Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systemscited by 2
  21. 2026Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judgescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Aggregate Disambiguation Systemscited by 0
  2. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0