Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

LLM judges now score outputs against rubrics, in benchmarks and as training rewards. Decision models return probabilities over permitted answers with a confidence score but no free text. Two such models are Jev and an open reproduction, Perplexity's Decider. We compare each with three flash-tier LLM judges on nine panels from seven benchmarks with human judgments, giving every judge identical criterion texts. The LLM judges run in two setups, reading a whole rubric at once or one criterion at a time. Our findings suggest both decision models can often stand in for them. The LLM judges cost 16 to 357 times as much and take 5 to 350 times as long, yet in both setups each decision model trails them significantly in at most 6 of 27 comparisons. The two kinds of judge err alike, agreeing more with one another than with the human-annotated judgment labels on ordinal criteria. Over 85% of flash-tier LLM verdicts on either decision model's most confident errors repeat them, where independent errors would give about half. Each decision model's confidence ranks its errors on most ordinal panels, so each is a natural first stage for a cascade deferring uncertain verdicts to an LLM judge. Yet with flash-tier fallbacks, such cascades gained a mere 2.7 points at best over the strongest single judge, even with oracle thresholds, suggesting they mainly save cost. A cascade needs a fallback that is right where its first stage errs, but the LLM judges are mostly wrong in the same places as Jev, and the Decider is already the strongest single judge on most panels. Even each vendor's higher-tier model repeats those errors, and as a fallback it lifted no cascade more than 1.7 points above both tiers' strongest single judge. Holding across setups, effort levels, and tiers, these findings suggest designing future decision models afresh to get right what LLM judges get wrong, and vice versa.