# JudgeMoE: Distributional Aggregation for LLM-as-a-Judge

arXiv: https://arxiv.org/abs/2610.07109 · HTML: https://mlsanity.com/paper/2610.07109/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Ministral 3](https://mlsanity.com/paper/2601.08584/) (2026)
- [Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles](https://mlsanity.com/paper/2401.00243/) (2023)
- [Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble](https://mlsanity.com/paper/2401.16635/) (2024)
- [Self-rationalization improves LLM as a fine-grained judge](https://mlsanity.com/paper/2410.05495/) (2024)
- [Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale](https://mlsanity.com/paper/2601.03444/) (2026)
- [An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability](https://mlsanity.com/paper/2506.13639/) (2025)
- [Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation](https://mlsanity.com/paper/2506.10403/) (2025)
- [FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge](https://mlsanity.com/paper/2602.06625/) (2026)

## Abstract (arXiv)

When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by +0.079. Applying the same configuration to six additional cells yields a +0.0393 mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p=0.0091. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.