mlsanityData to 7 Oct 2026

Small Language Models as Judges for Rubric-Based Reinforcement Learning

arXiv 2608.30005 ↗First posted 30 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7× more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

28 of 43 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2024A Survey on LLM-as-a-Judgecited by 1,384
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  10. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  11. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  12. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
16 more
  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  5. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  6. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  7. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  8. 2025Atla Selene Mini: A General Purpose Evaluation Modelcited by 27
  9. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  10. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  11. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 19
  12. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 17
  13. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 10
  14. 2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetrycited by 8
  15. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  16. 2026BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Rubric Anchors
  2. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  3. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  4. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models
  5. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
  6. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
  7. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  8. 2026Step-wise Rubric Rewards for LLM Reasoning