mlsanityData to 14 Oct 2026

Learning the Reward Function for a Misspecified Model

arXiv 1801.09624 ↗First posted 29 Jan 2018cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In model-based reinforcement learning it is typical to decouple the problems of learning the dynamics model and learning the reward function. However, when the dynamics model is flawed, it may generate erroneous states that would never occur in the true environment. It is not clear a priori what value the reward function should assign to such states. This paper presents a novel error bound that accounts for the reward model's behavior in states sampled from the model. This bound is used to extend the existing Hallucinated DAgger-MC algorithm, which offers theoretical performance guarantees in deterministic MDPs that do not assume a perfect model can be learned. Empirically, this approach to reward learning can yield dramatic improvements in control performance when the dynamics model is flawed.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 9 of its 36 references that are in the map. First posted on arXiv 29 Jan 2018.

Builds on

9 of 36 references.

References · 9

  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  2. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  3. 2016Value Iteration Networkscited by 160
  4. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  5. 2017Value Prediction Networkcited by 117
  6. 2016The Predictron: End-To-End Learning and Planningcited by 93
  7. 2017Recurrent Environment Simulatorscited by 68
  8. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  9. 2016Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Gamescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 21
  2. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  3. 2022Reward-Predictive Clusteringcited by 0
  4. 2026On Training in Imaginationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning
  2. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  3. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
  4. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
  5. 2025Model-Based Reinforcement Learning Under Confounding
  6. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  7. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
  8. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks