Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation

arXiv 2607.27984 ↗First posted 30 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agentic systems generate outputs faster than human review. We contrast two LLM evaluator specialization strategies: specialized judge weights, or rule-based deferral policies for safe judgment acceptance. On 99,952 rubric-conditioned samples, correct rubrics improve accuracy by 2.11 points, while incorrect rubrics reduce performance by 2.66 points. Splitting training data across eight criterion-specific LoRA experts lowers accuracy by 10.05 points and reduces 5% error-bound coverage from 24.44% to 5.43%. This loss is independent of model size and training settings, with most performance recoverable by warm-starting experts from a unified judge. Sweeping data budgets confirms scratch expert specialization yields no empirical gains. Warm-started splits appear competitive with unified models, yet under limited data, unified training outperforms split training, with specialization beneficial only after unified training plateaus. Results hold on HealthBench, where physician rubrics improve accuracy while flawed rubrics degrade performance. Unlike weight specialization, deferral policies enable efficient evaluation. On RewardBench 2, lightweight deferral heads form a 0.6B-4B-8B reward cascade with no core scoring modification. Across 20 splits, the cascade achieves 89.40% accuracy versus 84.75% for a standalone 8B judge at 41.5% compute, satisfying 95% risk constraints. Margin-based deferral matches accuracy at far higher compute cost. The design generalizes across models, improving Tulu-3-8B and Skywork-8B performance with a lightweight DeBERTa frontend. We derive simple, robust evaluator design rules: unify judgment training or warm-start split models, and use audited deferral cascades for low-cost, reliable LLM evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 19 of its 26 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

19 of 26 references.

Closest ancestors · 1

Same direction, newest first

2026AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning22 Mar

Other references · 18

2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
2017Selective Classification for Deep Neural Networkscited by 852
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
2020Consistent Estimators for Learning to Defer to an Expertcited by 271
2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105

6 more

2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
2025Gatekeeper: Improving Model Cascades Through Confidence Tuningcited by 10
2025Quantitative LLM Judgescited by 6
2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way