mlsanityData to 7 Oct 2026

Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy

arXiv 2603.15646 ↗First posted 4 Mar 2026cs.LG cs.AI cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reinforcement Learning with Rubric Rewards (RLRR) is a framework that extends conventional reinforcement learning from human feedback (RLHF) and verifiable rewards (RLVR) by replacing scalar preference signals with structured, multi-dimensional, contextual rubric-based evaluations. However, existing approaches in RLRR are limited to linearly compressing vector rewards into a scalar reward with a fixed weightings, which is sensitive to artificial score design and fails to capture correlations among reward dimensions. To overcome the limitations of reward aggregation, this work proposes Alternating Reinforcement Learning with Rubric Rewards (ARL-RR), a framework that eliminates the need for a fixed scalarization by optimizing one semantic rubric meta-class at a time. Theoretically, we show that reward aggregation induces a variance contraction effect, which helps explain the performance gains. We further introduce a lightweight, search-based adaptation procedure that selects the next meta-class dynamically based on task performance, enabling the policy to emphasize critical objectives and thereby improve the model performance. Empirically, our experiments on the HealthBench dataset with experts annotations demonstrate that ARL-RR uniformly outperforms scalarized methods in both model performance and training efficiency across different model scales (1.7B, 4B, 8B, and 14B).

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 57 references that are in the map. First posted on arXiv 4 Mar 2026.

Builds on

47 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training2 Feb
  2. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling20 Oct
  3. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
32 more
  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  2. 2025Group Sequence Policy Optimizationcited by 788
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  7. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  8. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  9. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  10. 2025Reinforcement Learning with Rubric Anchorscited by 108
  11. 2024Rule Based Rewards for Language Model Safetycited by 106
  12. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 89
  13. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  14. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  15. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  16. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  17. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  18. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  19. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  20. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-trainingcited by 45
  21. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  22. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  23. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learningcited by 30
  24. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  25. 2025Language Models that Think, Chat Bettercited by 23
  26. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
  27. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  28. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 17
  29. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  30. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 9
  31. 2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Modelscited by 9
  32. 2025AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Predictioncited by 2
  2. 2026Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventionscited by 1
  3. 2026ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answeringcited by 0
  4. 2026APTER: Adaptive Post-Training with Expert-Grounded Rubricscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
  3. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  4. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
  5. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models
  6. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
  7. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  8. 2026Reinforcement Learning with Robust Rubric Rewards