mlsanityData to 7 Oct 2026

Understanding Learned Reward Functions

arXiv 2012.05862 ↗First posted 10 Dec 2020cs.LGcited by 37

Not on the map. Why?

Cited by 26 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many real-world tasks, it is not possible to procedurally specify an RL agent's reward function. In such cases, a reward function must instead be learned from interacting with and observing humans. However, current techniques for reward learning may fail to produce reward functions which accurately reflect user preferences. Absent significant advances in reward learning, it is thus important to be able to audit learned reward functions to verify whether they truly capture user preferences. In this paper, we investigate techniques for interpreting learned reward functions. In particular, we apply saliency methods to identify failure modes and predict the robustness of reward functions. We find that learned reward functions often implement surprising algorithms that rely on contingent aspects of the environment. We also discover that existing interpretability techniques often attend to irrelevant changes in reward output, suggesting that reward interpretability may need significantly different methods from policy interpretability.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 10; 2024: 11; 2025: 6; 2026: 2.

37 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  6. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  7. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  9. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  10. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  11. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  12. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
25 more
  1. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  2. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  4. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11
  5. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  6. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  7. 2021Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functionscited by 10
  8. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  9. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  10. 2022Preprocessing Reward Functions for Interpretabilitycited by 7
  11. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
  12. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  13. 2022Calculus on MDPs: Potential Shaping as a Gradientcited by 4
  14. 2022Skill-Based Reinforcement Learning with Intrinsic Reward Matchingcited by 4
  15. 2023Interpreting Learned Feedback Patterns in Large Language Modelscited by 4
  16. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  17. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  18. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learningcited by 4
  19. 2025Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Modelscited by 4
  20. 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoffcited by 3
  21. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  22. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  23. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedbackcited by 0
  24. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?cited by 0
  25. 2026Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classificationcited by 0