mlsanityData to 14 Oct 2026

R3: Robust Rubric-Agnostic Reward Models

arXiv 2505.13388 ↗First posted 19 May 2025cs.CL cs.AI cs.LGcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are essential for aligning language model outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models are typically optimized for narrow objectives, limiting their generalizability to broader downstream tasks. Moreover, their scalar outputs are difficult to interpret without contextual reasoning. To address these limitations, we introduce R3, a novel reward modeling framework that is rubric-agnostic, generalizable across evaluation dimensions, and provides interpretable, reasoned score assignments. R3 enables more transparent and flexible evaluation of language models, supporting robust alignment with diverse human values and use cases. Our models, data, and code are available as open source at https://github.com/rubricreward/r3.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 63 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

55 of 63 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages16 May
  2. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scores14 Apr
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  6. 2024Critique-out-Loud Reward Models21 Aug
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  8. 2024Improving Reward Models with Synthetic Critiques31 May
  9. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  10. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 44

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
32 more
  1. 2025s1: Simple test-time scalingcited by 1,341
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  13. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  14. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  15. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  16. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  17. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  18. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  19. 2025Phi-4-reasoning Technical Reportcited by 151
  20. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  21. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 93
  22. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  23. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  24. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  25. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  26. 2024Self-Taught Evaluatorscited by 72
  27. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  28. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  29. 2023Evaluating Open-QA Evaluationcited by 35
  30. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  31. 2023Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Modelcited by 21
  32. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  3. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 30
  4. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  5. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  6. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 8
  7. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  8. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  9. 2025Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountabilitycited by 6
  10. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 6
  11. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  12. 2025SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languagescited by 4
11 more
  1. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  2. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  3. 2025IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesiancited by 1
  4. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  5. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  6. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  7. 2025Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languagescited by 0
  8. 2026CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluationcited by 0
  9. 2026Human-Centric Image Captioning with Subject-Centered Spatial Understandingcited by 0
  10. 2026Online Language Adaptive Sampling for Better Distributed Cross-lingual Gainscited by 0
  11. 2026Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  2. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  3. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  4. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  5. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  6. 2026Reward Models Inherit Value Biases from Pretraining
  7. 2024Transforming and Combining Rewards for Aligning Large Language Models
  8. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling