mlsanityData to 14 Oct 2026

Inverse Reward Design

arXiv 1711.02827 ↗First posted 8 Nov 2017cs.AI cs.LGcited by 249

Not on the map. Why?

Cited by 67 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.

Key facts. Cited by 249 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 4; 2020: 2; 2021: 42; 2022: 36; 2023: 34; 2024: 39; 2025: 51; 2026: 39.

249 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  5. 2023Reward Design with Language Modelscited by 261
  6. 2023Large Language Model Alignment: A Surveycited by 243
  7. 2022Defining and Characterizing Reward Hackingcited by 203
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  9. 2021Alignment of Language Agentscited by 168
  10. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  11. 2023Getting aligned on representational alignmentcited by 143
  12. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
38 more
  1. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  2. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  3. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  4. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  5. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  6. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  7. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 100
  8. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  9. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  10. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  11. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 86
  12. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  13. 2021Consequences of Misaligned AIcited by 80
  14. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  15. 2021On the Expressivity of Markov Rewardcited by 71
  16. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  17. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  18. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  19. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  20. 2019Multi-Agent Adversarial Inverse Reinforcement Learningcited by 60
  21. 2022Models of human preference for learning reward functionscited by 54
  22. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 53
  23. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  24. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  25. 2022Inferring Rewards from Language in Contextcited by 50
  26. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  27. 2019Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safetycited by 49
  28. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 38
  29. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 37
  30. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  31. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  32. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 35
  33. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  34. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  35. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  36. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 30
  37. 2021What are you optimizing for? Aligning Recommender Systems with Human Valuescited by 29
  38. 2020Bayesian Robust Optimization for Imitation Learningcited by 28

50 most cited of 249 All 249 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  2. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
  3. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
  4. 2025Reward Compatibility: A Framework for Inverse RL
  5. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience
  6. 2019Learning Human Objectives by Evaluating Hypothetical Behavior
  7. 2021Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
  8. 2024Multi Task Inverse Reinforcement Learning for Common Sense Reward