mlsanityData to 7 Oct 2026

Reward Models in Deep Reinforcement Learning: A Survey

arXiv 2506.15421 ↗First posted 18 Jun 2025cs.LG cs.AIcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In reinforcement learning (RL), agents continually interact with the environment and use the feedback to refine their behavior. To guide policy optimization, reward models are introduced as proxies of the desired objectives, such that when the agent maximizes the accumulated reward, it also fulfills the task designer's intentions. Recently, significant attention from both academic and industrial researchers has focused on developing reward models that not only align closely with the true objectives but also facilitate policy optimization. In this survey, we provide a comprehensive review of reward modeling techniques within the deep RL literature. We begin by outlining the background and preliminaries in reward modeling. Next, we present an overview of recent reward modeling approaches, categorizing them based on the source, the mechanism, and the learning paradigm. Building on this understanding, we discuss various applications of these reward modeling techniques and review methods for evaluating reward models. Finally, we conclude by highlighting promising research directions in reward modeling. Altogether, this survey includes both established and emerging methods, filling the vacancy of a systematic review of reward models in current literature.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 78 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

56 of 78 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd19 Nov
  2. 2024Hindsight PRIORs for Reward Learning from Human Preferences12 Apr
  3. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  4. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback21 Nov
  8. 2022Models of human preference for learning reward functions5 Jun
  9. 2022Preprocessing Reward Functions for Interpretability25 Mar
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
34 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  3. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  6. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  7. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  8. 2019Self-Supervised Exploration via Disagreementcited by 263
  9. 2017Count-Based Exploration with Neural Density Modelscited by 255
  10. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  12. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  13. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  14. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  15. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  16. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  17. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  18. 2021Discovering and Achieving Goals via World Modelscited by 131
  19. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  20. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  21. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  22. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  23. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  24. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  25. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  26. 2023Vision-Language Models as a Source of Rewardscited by 65
  27. 2020Quantifying Differences in Reward Functionscited by 62
  28. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  29. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  30. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  31. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
  32. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  33. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  34. 2020Regularized Inverse Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 18.

24 papers.

Sort:

By area:

  1. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  2. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
  3. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
  4. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  5. 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judgescited by 3
  6. 2025PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoningcited by 2
  7. 2026MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Modelscited by 2
  8. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  9. 2026Reward Prediction with Factorized World Statescited by 2
  10. 2026SteerRM: Debiasing Reward Models via Sparse Autoencoderscited by 2
  11. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  12. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
12 more
  1. 2025When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialoguecited by 1
  2. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  3. 2026D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Modelscited by 1
  4. 2025Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learningcited by 0
  5. 2025Monotonic Transformation Invariant Multi-task Learningcited by 0
  6. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learningcited by 0
  7. 2026Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deploymentcited by 0
  8. 2026Distributed Direct Preference Optimizationcited by 0
  9. 2026ReCal: Reward Calibration for RL-based LLM Routingcited by 0
  10. 2026PortraitGen: Exemplar-Driven GRPO with Dual-Reward Guidance for Photorealistic Portrait Generationcited by 0
  11. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0
  12. 2026Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrastcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning for Large Model: A Survey
  2. 2018Scalable agent alignment via reward modeling: a research direction
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
  5. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  6. 2026Deep Reinforcement Learning: From First Principles to Reasoning Models ↗
  7. 2025Reinforcement Learning from Human Feedback
  8. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards