mlsanityData to 14 Oct 2026

JudgeMoE: Distributional Aggregation for LLM-as-a-Judge

arXiv 2610.07109 ↗First posted 5 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by +0.079. Applying the same configuration to six additional cells yields a +0.0393 mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p=0.0091. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 47 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

13 of 47 references.

References · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  6. 2026Ministral 3cited by 114
  7. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  8. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 44
  9. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  10. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 20
  11. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 19
  12. 2025Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluationcited by 6
1 more
  1. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judgecited by 5

Built on by

Not cited yet.