mlsanityData to 7 Oct 2026

SLMJury: Can Small Language Models Judge as Well as Large Ones?

arXiv 2606.07810 ↗First posted 5 Jun 2026cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability. We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms: closed-ended binary correctness and open-ended quality scoring. We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks: eight closed-ended tasks spanning mathematical, scientific, and general reasoning (N=64,824 judgments per configuration), plus SummEval and MT-Bench for summarization and conversational scoring. We formalize judging as a budget-conditioned function and study five dimensions. Four findings emerge. (1) The overthinking effect is domain-dependent: for most judges quick 10-token verdicts match or beat extended reasoning on mathematical judging (by 2-7% where they help), while reasoning wins on general tasks by up to 23%. (2) Domain generalization separates model families, with math-to-general accuracy gaps ranging from under 10% to nearly 40%. (3) Closed-ended and open-ended judging draw on different capabilities: the best binary judge (Phi-4) drops to rank 9 on MT-Bench, while reasoning-trained models invert this ordering. (4) Under the Reflect-Critique-Refine (RCR) debate protocol, multi-agent debate degrades accuracy across all tested configurations, whereas the top judges resist six adversarial personas with <=0.55% variance. Reliable automated evaluation does not require large proprietary models, yet no single SLM dominates. The leaderboard is available at https://anishh15.github.io/SLMJury/, and our framework code and pip package are publicly available at https://github.com/anishh15/SLMJury and https://pypi.org/project/slmjury/.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

46 of 64 references.

References · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  12. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
34 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  5. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  6. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  7. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  8. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  11. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  12. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  13. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  14. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 102
  15. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 63
  16. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  17. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  18. 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneitycited by 43
  19. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMscited by 35
  20. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  21. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  22. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decodingcited by 22
  23. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  24. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 20
  25. 2025Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detectorcited by 14
  26. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  27. 2025COSMosFL: Ensemble of Small Language Models for Fault Localisationcited by 3
  28. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustnesscited by 3
  29. 2026The Persona Paradox: Medical Personas as Behavioral Priors in Clinical Language Modelscited by 3
  30. 2026Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluationcited by 3
  31. 2025Efficient LLM Safety Evaluation through Multi-Agent Debatecited by 2
  32. 2025JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluationcited by 2
  33. 2026Small Reward Models via Backward Inferencecited by 2
  34. 2026Improving Code Generation via Small Language Model-as-a-judgecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluationcited by 0
  2. 2026PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluatorscited by 0
  3. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
  2. 2025Towards Reasoning Ability of Small Language Models
  3. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  4. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  5. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
  6. 2024Are Your LLMs Capable of Stable Reasoning?
  7. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  8. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models