mlsanityData to 14 Oct 2026

Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning

arXiv 2509.25534 ↗First posted 19 Sep 2025cs.CL cs.AI cs.LGcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Open-ended evaluation is essential for deploying large language models in real-world settings. In studying HealthBench, we observe that using the model itself as a grader and generating rubric-based reward signals substantially improves reasoning performance. Remarkably, the trained model also becomes a stronger grader. Motivated by this, we introduce Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning, a lightweight framework that enables faster and more resource-efficient training while surpassing baselines. Remarkably, on Qwen3-32B, training with just the 4000-sample HealthBench Easy subset is sufficient to obtain a model that exceeds GPT-5 on HealthBench Hard. Incorporating a small amount of teacher-graded data further enhances performance for less capable models.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 20 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

19 of 20 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024CREAM: Consistency Regularized Self-Rewarding Language Models16 Oct
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
3 more
  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2025Reinforcement Learning with Rubric Anchorscited by 113
  3. 2025Re-evaluating Open-ended Evaluation of Large Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  2. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  3. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  4. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 6
  5. 2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4
  6. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  7. 2026From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learningcited by 0
  8. 2026Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signalscited by 0
  9. 2026Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewardscited by 0
  10. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learningcited by 0
  11. 2026Evaluating Rubric Generation with Interventional Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  2. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
  3. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
  4. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  5. 2026Rubrics as Privileged Information for Open-Ended Generation
  6. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
  7. 2026Health-SCORE: Towards Scalable Rubrics for Improving Health-LLMs
  8. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation