# How Many Humans Are 32 LLM Judges Worth?

arXiv: https://arxiv.org/abs/2609.21277 · HTML: https://mlsanity.com/paper/2609.21277/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2026.

## Built on by (most cited first, 1 of 1)

- [Gaussian Process Bandits for AI System Optimization with Selectively Verified Feedback](https://mlsanity.com/paper/2610.17760/) (2026)

## Abstract (arXiv)

A panel's human-equivalent size is target-specific. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks yields two distinct effective sizes: distributional-error matching gives nuMSE=2.304, 3.750, and 3.445, whereas spectral matching gives nuH=4.242, 6.459, and 6.499, a gap of 1.72--1.89×; a binary-error diagnostic credits the same panels with only 1.971--2.227 effective votes. Extrapolating the distributional-error curve at fixed squared mean residual, mean member variance, and normalized mean covariance gives asymptotes of 2.392, 3.990, and 3.655, with 32 judges already reaching 94.0--96.3%. An exact spectral identity explains the gap: error depends on member energy and on the orientation of residual variation relative to averaging, information that the participation ratio (PR) discards. A realizable hard-label construction confirms that higher spectral diversity can coexist with worse distribution recovery even under equal member energies and nonnegative correlations, and the consensus direction retains γco=43.8%, 33.7%, and 35.9% of centered residual variance. An external check on CC-1000, a 1,000-item Civil Comments subset with a different panel, gives nuH=2.84. For panel choice, we establish an existence result and one feasible path: exhaustive enumeration at k∈5,7 shows that panels beating the accuracy-top-k baseline on both accuracy and nuH always exist, and greedily swapping at most two members reaches 24.8--56.0% higher nuH at 0.10--1.10 percentage points higher accuracy. Our dataset and code are available at https://github.com/Chao1208/32judges-votes.