mlsanityData to 7 Oct 2026

Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training

arXiv 2602.01511 ↗First posted 2 Feb 2026cs.CL cs.LGcited by 45

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 69 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

60 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling20 Oct
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment9 Oct
  3. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training25 Sep

Other references · 57

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
38 more
  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  5. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  9. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  10. 2024Iterative Reasoning Preference Optimizationcited by 244
  11. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  12. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  13. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  14. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  15. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  16. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  17. 2024Generative Reward Modelscited by 148
  18. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  19. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  20. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  21. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  22. 2025Reinforcement Learning with Rubric Anchorscited by 108
  23. 2024Critique-out-Loud Reward Modelscited by 105
  24. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  25. 2024How to Evaluate Reward Models for RLHFcited by 84
  26. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  27. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  28. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  29. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  30. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  31. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  32. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  33. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  34. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 42
  35. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  36. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  37. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimizationcited by 21
  38. 2021Amortized Variational Inference for Simple Hierarchical Modelscited by 19

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 45.

45 papers.

Sort:

By area:

  1. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  2. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 12
  3. 2026EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learningcited by 11
  4. 2026Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generationcited by 7
  5. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 7
  6. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  7. 2026EMPA: Evaluating Persona-Aligned Empathy as a Processcited by 4
  8. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  9. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 4
  10. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  11. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  12. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Modelscited by 3
33 more
  1. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3
  2. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 3
  3. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  4. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 2
  5. 2026CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubricscited by 2
  6. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  7. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 2
  8. 2026RUBAS: Rubric-Based Reinforcement Learning for Agent Safetycited by 2
  9. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluationcited by 2
  10. 2026AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignmentcited by 1
  11. 2026Prompt-Level Reward Specifications for Open-Ended Post-Trainingcited by 1
  12. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  13. 2026QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewardscited by 1
  14. 2026Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubricscited by 1
  15. 2026Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoningcited by 1
  16. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Modelscited by 1
  17. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 1
  18. 2026RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modelingcited by 1
  19. 2026Document Reconstruction Unlocks Scalable Long-Context RLVRcited by 0
  20. 2026LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Modelscited by 0
  21. 2026Co-ReAct: Rubrics as Step-Level Collaborators for ReAct Agentscited by 0
  22. 2026GrowLoop: Self-Evolving Conversation Evaluation Seeded by Humancited by 0
  23. 2026Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modelingcited by 0
  24. 2026Personalization as Inverse Planning: Learning Latent Design Intents for Agentic Slide Generation via Structural Denoisingcited by 0
  25. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 0
  26. 2026RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Reviewcited by 0
  27. 2026PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesiscited by 0
  28. 2026ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agentscited by 0
  29. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 0
  30. 2026RankBuffer: Efficient Ranking-Based Rewards for Open-Ended Generationcited by 0
  31. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 0
  32. 2026Training LLM Judges from Language Feedback via Position-Selective Self-Distillationcited by 0
  33. 2026How to post-train on a surrogate: Envelope sampling mitigates reward hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  2. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
  3. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  4. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  5. 2026Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
  6. 2026Reward Hacking in Rubric-Based Reinforcement Learning
  7. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
  8. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning