mlsanityData to 7 Oct 2026

Quantifying Differences in Reward Functions

arXiv 2006.13900 ↗First posted 24 Jun 2020cs.LG cs.AI stat.MLcited by 62

Not on the map. Why?

Cited by 22 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For many tasks, the reward function is inaccessible to introspection or too complex to be specified procedurally, and must instead be learned from user data. Prior work has evaluated learned reward functions by evaluating policies optimized for the learned reward. However, this method cannot distinguish between the learned reward function failing to reflect user preferences and the policy optimization process failing to optimize the learned reward. Moreover, this method can only tell us about behavior in the evaluation environment, but the reward may incentivize very different behavior in even a slightly different deployment environment. To address these problems, we introduce the Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly, without a policy optimization step. We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy. Furthermore, we find EPIC can be efficiently approximated and is more robust than baselines to the choice of coverage distribution. Finally, we show that EPIC distance bounds the regret of optimal policies even under different transition dynamics, and we confirm empirically that it predicts policy training success. Our source code is available at https://github.com/HumanCompatibleAI/evaluating-rewards.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 12; 2024: 12; 2025: 10; 2026: 10.

62 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  3. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  4. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  5. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  6. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  8. 2023Goodhart's Law in Reinforcement Learningcited by 37
  9. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  10. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  11. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  12. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
38 more
  1. 2023On The Fragility of Learned Reward Functionscited by 21
  2. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
  3. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  4. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  5. 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
  6. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  7. 2022Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunitiescited by 12
  8. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  9. 2021Uncertain Decisions Facilitate Better Preference Learningcited by 10
  10. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  11. 2025Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitionerscited by 10
  12. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferencescited by 8
  13. 2023Designing Fiduciary Artificial Intelligencecited by 8
  14. 2022Preprocessing Reward Functions for Interpretabilitycited by 7
  15. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  16. 2021A Taxonomy of Similarity Metrics for Markov Decision Processescited by 6
  17. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  18. 2022Calculus on MDPs: Potential Shaping as a Gradientcited by 4
  19. 2022Skill-Based Reinforcement Learning with Intrinsic Reward Matchingcited by 4
  20. 2023Interpreting Learned Feedback Patterns in Large Language Modelscited by 4
  21. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  22. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  23. 2021Inverse Reinforcement Learning with Explicit Policy Estimatescited by 3
  24. 2024Transformable Gaussian Reward Function for Socially-Aware Navigation with Deep Reinforcement Learningcited by 3
  25. 2025Bootstrapped Reward Shapingcited by 3
  26. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
  27. 2025ADPO: Anchored Direct Preference Optimizationcited by 3
  28. 2021LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learningcited by 2
  29. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  30. 2024Cooperation and Control in Delegation Gamescited by 2
  31. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  32. 2026Reward Prediction with Factorized World Statescited by 2
  33. 2024A Generalized Acquisition Function for Preference-based Reward Learningcited by 1
  34. 2024Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Reviewcited by 1
  35. 2024Preserving the Privacy of Reward Functions in MDPs through Deceptioncited by 1
  36. 2024Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generationcited by 1
  37. 2025Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learningcited by 1
  38. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1

50 most cited of 62 All 62 (JSON)