# Active Inverse Reward Design

arXiv: https://arxiv.org/abs/1809.03060 · HTML: https://mlsanity.com/paper/1809.03060/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2018.

## Built on by (most cited first, 12 of 12)

- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Direct Behavior Specification via Constrained Reinforcement Learning](https://mlsanity.com/paper/2112.12228/) (2021)
- [Choice Set Misspecification in Reward Inference](https://mlsanity.com/paper/2101.07691/) (2021)
- [Active Reward Learning from Multiple Teachers](https://mlsanity.com/paper/2303.00894/) (2023)
- [Uncertain Decisions Facilitate Better Preference Learning](https://mlsanity.com/paper/2106.10394/) (2021)
- [Combining Reward Information from Multiple Sources](https://mlsanity.com/paper/2103.12142/) (2021)
- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [Avoiding Catastrophe in Online Learning by Asking for Help](https://mlsanity.com/paper/2402.08062/) (2024)
- [Iterative Reward Shaping using Human Feedback for Correcting Reward Misspecification](https://mlsanity.com/paper/2308.15969/) (2023)
- [Constrained Group Relative Policy Optimization](https://mlsanity.com/paper/2602.05863/) (2026)
- [Effective Reward Specification in Deep Reinforcement Learning](https://mlsanity.com/paper/2412.07177/) (2024)

## Abstract (arXiv)

Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus we can actively select queries for maximum informativeness about the true reward. In contrast to approaches asking the designer for optimal behavior, this allows us to gather additional information by eliciting preferences between suboptimal behaviors. After each query, we need to update the posterior over the true reward function from observing the proxy reward function chosen by the designer. The recently proposed Inverse Reward Design (IRD) enables this. Our approach substantially outperforms IRD in test environments. In particular, it can query the designer about interpretable, linear reward functions and still infer non-linear ones.