mlsanityData to 14 Oct 2026

RewardAnything: Generalizable Principle-Following Reward Models

arXiv 2506.03637 ↗First posted 4 Jun 2025cs.CL cs.AI cs.LGcited by 28

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward Models, essential for guiding Large Language Model optimization, are typically trained on fixed preference datasets, resulting in rigid alignment to single, implicit preference distributions. This prevents adaptation to diverse real-world needs-from conciseness in one task to detailed explanations in another. The standard practice of collecting task-specific preference data and retraining reward models is resource-intensive, often producing biased rewards, and limits practical application. We introduce generalizable, principle-following reward models. We propose that RMs should understand and adhere to dynamically provided natural language specifications of reward principles, similar to instruction-following in LLMs. To measure this capability, we develop RABench, a comprehensive benchmark for RMs focusing on generalization across diverse principles. Evaluations on RABench reveal poor generalization of current RMs. As a solution, we present RewardAnything, a novel RM designed and trained to explicitly follow natural language principles. We achieve SotA performance with RewardAnything in traditional RM benchmark simply by specifying a well-defined principle, and results on RABench show we excel in adapting to novel principles without retraining. Furthermore, RewardAnything integrates seamlessly with existing RLHF methods and we show by a case study on how to automatically and efficiently align LLMs with only natural language principles.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 84 of its 99 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

84 of 99 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization19 May
  2. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  3. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  4. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems26 Feb
  5. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  6. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  8. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  9. 2024How to Evaluate Reward Models for RLHF18 Oct
  10. 2024Generative Reward Models2 Oct
  11. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep
  12. 2024Critique-out-Loud Reward Models21 Aug
10 more
  1. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  3. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  4. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  12. 2024Qwen2.5 Technical Reportcited by 4,942
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  13. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  14. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  15. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  16. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  17. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  18. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  19. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  20. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  21. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  22. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  23. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  24. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  25. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  26. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  27. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  28. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  29. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  30. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  31. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  32. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  33. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  34. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  35. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  36. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 42
  37. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  38. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 18.

28 papers.

Sort:

By area:

  1. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  2. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  3. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  4. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  5. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  6. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  7. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 11
  8. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 10
  9. 2025Reasoning-Enhanced Large Language Models for Molecular Property Predictioncited by 9
  10. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 8
  11. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  12. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
16 more
  1. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 6
  2. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  3. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  4. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 3
  5. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 3
  6. 2026Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agentscited by 3
  7. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  8. 2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Modelscited by 1
  9. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  10. 2026Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubricscited by 1
  11. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 1
  12. 2025MetaLint: Easy-to-Hard Generalization for Code Lintingcited by 0
  13. 2026Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarizationcited by 0
  14. 2026UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigmscited by 0
  15. 2026F2DR: A Fine-Grained Full-Pipeline Reward Framework for DeepSearch Workflowscited by 0
  16. 2026RM-OPD: On-Policy Distillation for Generative Reward Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  2. 2026Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions
  3. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
  4. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  5. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  6. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  7. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  8. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling