mlsanityData to 7 Oct 2026

Iterative Reward Shaping using Human Feedback for Correcting Reward Misspecification

arXiv 2308.15969 ↗First posted 30 Aug 2023cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

A well-defined reward function is crucial for successful training of an reinforcement learning (RL) agent. However, defining a suitable reward function is a notoriously challenging task, especially in complex, multi-objective environments. Developers often have to resort to starting with an initial, potentially misspecified reward function, and iteratively adjusting its parameters, based on observed learned behavior. In this work, we aim to automate this process by proposing ITERS, an iterative reward shaping approach using human feedback for mitigating the effects of a misspecified reward function. Our approach allows the user to provide trajectory-level feedback on agent's behavior during training, which can be integrated as a reward shaping signal in the following training iteration. We also allow the user to provide explanations of their feedback, which are used to augment the feedback and reduce user effort and feedback frequency. We evaluate ITERS in three environments and show that it can successfully correct misspecified reward functions.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 20 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

16 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 14

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  3. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  6. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  7. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  8. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  9. 2019Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitationscited by 50
  10. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  11. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  12. 2020FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedbackcited by 15
2 more
  1. 2018Active Inverse Reward Designcited by 12
  2. 2023Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shapingcited by 3
  2. 2024Tell me why: Training preferences-based RL with human preferences and step-level explanationscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Behavior Alignment via Reward Function Optimization
  2. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  3. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  4. 2023REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback
  5. 2026Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
  6. 2025Reward Shaping to Mitigate Reward Hacking in RLHF
  7. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  8. 2020Reward Shaping for Human Learning via Inverse Reinforcement Learning