# Robust Reward Design for Markov Decision Processes

arXiv: https://arxiv.org/abs/2406.05086 · HTML: https://mlsanity.com/paper/2406.05086/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 7 of 7)

- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [A unified view of entropy-regularized Markov decision processes](https://mlsanity.com/paper/1705.07798/) (2017)
- [PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2308.02585/) (2023)
- [Principal-Agent Reward Shaping in MDPs](https://mlsanity.com/paper/2401.00298/) (2023)
- [Admissible Policy Teaching through Reward Design](https://mlsanity.com/paper/2201.02185/) (2022)
- [Robust Stackelberg Equilibria](https://mlsanity.com/paper/2304.14990/) (2023)
- [Generalized Principal-Agency: Contracts, Information, Games and Beyond](https://mlsanity.com/paper/2209.01146/) (2022)

## Built on by (most cited first, 1 of 1)

- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)

## Abstract (arXiv)

The problem of reward design examines the interaction between a leader and a follower, where the leader aims to shape the follower's behavior to maximize the leader's payoff by modifying the follower's reward function. Current approaches to reward design rely on an accurate model of how the follower responds to reward modifications, which can be sensitive to modeling inaccuracies. To address this issue of sensitivity, we present a solution that offers robustness against uncertainties in modeling the follower, including 1) how the follower breaks ties in the presence of nonunique best responses, 2) inexact knowledge of how the follower perceives reward modifications, and 3) bounded rationality of the follower. Our robust solution is guaranteed to exist under mild conditions and can be obtained numerically by solving a mixed-integer linear program. Numerical experiments on multiple test cases demonstrate that our solution improves robustness compared to the standard approach without incurring significant additional computing costs.