# Small Language Models as Judges for Rubric-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.30005 · HTML: https://mlsanity.com/paper/2608.30005/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [RLTF: Reinforcement Learning from Unit Test Feedback](https://mlsanity.com/paper/2307.04349/) (2023)
- [DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research](https://mlsanity.com/paper/2511.19399/) (2025)
- [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024)
- [RubricBench: Aligning Model-Generated Rubrics with Human Standards](https://mlsanity.com/paper/2603.01562/) (2026)
- [Atla Selene Mini: A General Purpose Evaluation Model](https://mlsanity.com/paper/2501.17195/) (2025)
- [Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation](https://mlsanity.com/paper/2303.03608/) (2023)
- [SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks](https://mlsanity.com/paper/2507.01001/) (2025)
- [RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following](https://mlsanity.com/paper/2603.25133/) (2026)
- [Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judge](https://mlsanity.com/paper/2505.19477/) (2025)
- [IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation](https://mlsanity.com/paper/2603.04738/) (2026)
- [Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry](https://mlsanity.com/paper/2601.22588/) (2026)
- [BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation](https://mlsanity.com/paper/2604.09497/) (2026)

## Built on by (most cited first, 2 of 2)

- [Efficient Resource-Shiftable RL Post-training with Generative Rewards](https://mlsanity.com/paper/2610.16866/) (2026)
- [Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.19176/) (2026)

## Abstract (arXiv)

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7× more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.