mlsanityData to 14 Oct 2026

A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future

arXiv 2504.12328 ↗First posted 12 Apr 2025cs.CL cs.AIcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward Model (RM) has demonstrated impressive potential for enhancing Large Language Models (LLM), as RM can serve as a proxy for human preferences, providing signals to guide LLMs' behavior in various tasks. In this paper, we provide a comprehensive overview of relevant research, exploring RMs from the perspectives of preference collection, reward modeling, and usage. Next, we introduce the applications of RMs and discuss the benchmarks for evaluation. Furthermore, we conduct an in-depth analysis of the challenges existing in the field and dive into the potential research directions. This paper is dedicated to providing beginners with a comprehensive introduction to RMs and facilitating future studies. The resources are publicly available at githubfootnotehttps://github.com/JLZhong23/awesome-reward-models.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 196 of its 226 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

196 of 226 references.

Closest ancestors · 59

Same direction, newest first

  1. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking31 Jan
  2. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment16 Jan
  3. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  6. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings20 Oct
  7. 2024How to Evaluate Reward Models for RLHF18 Oct
  8. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  9. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?8 Oct
  10. 2024Generative Reward Models2 Oct
  11. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Data1 Oct
  12. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
38 more
  1. 2024Reward-Robust RLHF in LLMs18 Sep
  2. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  3. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models11 Sep
  4. 2024Reinforcement Learning for LLM Post-Training: A Survey23 Jul
  5. 2024Towards Comprehensive Preference Data Collection for Reward Modeling24 Jun
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  7. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  8. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  9. 2024Improving Reward Models with Synthetic Critiques31 May
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  11. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  12. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  13. 2024Prior Constraints-based Reward Model Training for Aligning Large Language Models1 Apr
  14. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  15. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  16. 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling2 Mar
  17. 2024Bayesian Reward Models for LLM Alignment20 Feb
  18. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  19. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback4 Feb
  20. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  21. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  22. 2024Self-Rewarding Language Models18 Jan
  23. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  24. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  25. 2023Nash Learning from Human Feedback1 Dec
  26. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift21 Nov
  27. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  28. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  29. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  30. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  31. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback8 Aug
  32. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  33. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  34. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  35. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  36. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  37. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  38. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr

50 newest of 59 All 59 (JSON)

Other references · 137

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2020Learning to summarize from human feedbackcited by 2,588
38 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  14. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  15. 2020A Survey of Deep Active Learningcited by 554
  16. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  17. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  18. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  19. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  20. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  21. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  22. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  23. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  24. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  25. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  26. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  27. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  28. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  29. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  30. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  31. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  32. 2021Ensemble deep learning: A reviewcited by 284
  33. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  34. 2023AI Alignment: A Comprehensive Surveycited by 263
  35. 2023Reward Design with Language Modelscited by 261
  36. 2024Iterative Reasoning Preference Optimizationcited by 246
  37. 2023Large Language Model Alignment: A Surveycited by 243
  38. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242

50 most cited of 137 All 137 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 33; 2026: 20.

53 papers.

Sort:

By area:

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
  4. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  5. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  6. 2025Reward Reasoning Modelcited by 47
  7. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  8. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  9. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  10. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 32
  11. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 29
  12. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
38 more
  1. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  2. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  3. 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 18
  4. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 17
  5. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  6. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
  7. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  8. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 12
  9. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  10. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  11. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  12. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  13. 2025Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidencecited by 8
  14. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  15. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  16. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 8
  17. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  18. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6
  19. 2025Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Modelscited by 6
  20. 2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
  21. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  22. 2025TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoningcited by 5
  23. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  24. 2026Interpreting and Controlling LLM Reasoning through Integrated Policy Gradientcited by 3
  25. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  26. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 3
  27. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 3
  28. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillationcited by 2
  29. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  30. 2025Uncertainty-aware Reward Design Processcited by 1
  31. 2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
  32. 2026Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modelingcited by 1
  33. 2026Learning Ordinal Probabilistic Reward from Preferencescited by 1
  34. 2026StoryAlign: Evaluating and Training Reward Models for Story Generationcited by 1
  35. 2025RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoningcited by 0
  36. 2025Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewardscited by 0
  37. 2025Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLMcited by 0
  38. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferencescited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BaseReward: A Strong Baseline for Multimodal Reward Model
  2. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
  3. 2025RewardBench 2: Advancing Reward Model Evaluation
  4. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  5. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  6. 2026Reward Models Inherit Value Biases from Pretraining
  7. 2023Tool-Augmented Reward Modeling
  8. 2025Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling