mlsanityData to 7 Oct 2026

On The Fragility of Learned Reward Functions

arXiv 2301.03652 ↗First posted 9 Jan 2023cs.LG cs.AIcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward functions are notoriously difficult to specify, especially for tasks with complex goals. Reward learning approaches attempt to infer reward functions from human feedback and preferences. Prior works on reward learning have mainly focused on the performance of policies trained alongside the reward function. This practice, however, may fail to detect learned rewards that are not capable of training new policies from scratch and thus do not capture the intended behavior. Our work focuses on demonstrating and studying the causes of these relearning failures in the domain of preference-based reward learning. We demonstrate with experiments in tabular and continuous control environments that the severity of relearning failures can be sensitive to changes in reward model design and the trajectory dataset composition. Based on our findings, we emphasize the need for more retraining-based evaluations in the literature.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 46 references that are in the map. First posted on arXiv 9 Jan 2023.

Builds on

26 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  5. 2019Positive-Unlabeled Reward Learning1 Nov
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 217
  11. 2022Defining and Characterizing Reward Hackingcited by 184
  12. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
6 more
  1. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  2. 2021Consequences of Misaligned AIcited by 79
  3. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  4. 2020Quantifying Differences in Reward Functionscited by 62
  5. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  6. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 12; 2025: 5.

21 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  3. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  4. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  5. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  6. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  7. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  8. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  9. 2023Compositional preference models for aligning LMscited by 25
  10. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  11. 2024Learning to Watermark LLM-generated Text via Reinforcement Learningcited by 19
  12. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
9 more
  1. 2024Generalizing Reward Modeling for Out-of-Distribution Preference Learningcited by 6
  2. 2024Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agentscited by 6
  3. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
  4. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Trainingcited by 5
  5. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  6. 2024Dishonesty in Helpful and Harmless Alignmentcited by 3
  7. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
  8. 2025Establishing Reliability Metrics for Reward Models in Large Language Modelscited by 1
  9. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hackingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences
  2. 2026The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning
  3. 2024Hindsight PRIORs for Reward Learning from Human Preferences
  4. 2022Models of human preference for learning reward functions
  5. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
  6. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  7. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback
  8. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward