# Computing Policies That Account For The Effects Of Human Agent Uncertainty During Execution In Markov Decision Processes

arXiv: https://arxiv.org/abs/2109.07436 · HTML: https://mlsanity.com/paper/2109.07436/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 3 of 3)

- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Solving POMDPs by Searching the Space of Finite Policies](https://mlsanity.com/paper/1301.6720/) (2013)
- [A Bayesian Approach to Identifying Representational Errors](https://mlsanity.com/paper/2103.15171/) (2021)

## Built on by (most cited first, 3 of 3)

- [Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08738/) (2023)
- [Data Driven Reward Initialization for Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08733/) (2023)
- [Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments](https://mlsanity.com/paper/2302.14208/) (2023)

## Abstract (arXiv)

When humans are given a policy to execute, there can be policy execution errors and deviations in policy if there is uncertainty in identifying a state. This can happen due to the human agent's cognitive limitations and/or perceptual errors. So an algorithm that computes a policy for a human to execute ought to consider these effects in its computations. An optimal Markov Decision Process (MDP) policy that is poorly executed (because of a human agent) maybe much worse than another policy that is suboptimal in the MDP, but considers the human-agent's execution behavior. In this paper we consider two problems that arise from state uncertainty; these are erroneous state-inference, and extra-sensing actions that a person might take as a result of their uncertainty. We present a framework to model the human agent's behavior with respect to state uncertainty, and can be used to compute MDP policies that accounts for these problems. This is followed by a hill climbing algorithm to search for good policies given our model of the human agent. We also present a branch and bound algorithm which can find the optimal policy for such problems. We show experimental results in a Gridworld domain, and warehouse-worker domain. Finally, we present human-subject studies that support our human model assumptions.