# Inverse Policy Evaluation for Value-based Sequential Decision-making

arXiv: https://arxiv.org/abs/2008.11329 · HTML: https://mlsanity.com/paper/2008.11329/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 1 of 1)

- [The Geometry of Robust Value Functions](https://mlsanity.com/paper/2201.12929/) (2022)

## Abstract (arXiv)

Value-based methods for reinforcement learning lack generally applicable ways to derive behavior from a value function. Many approaches involve approximate value iteration (e.g., Q-learning), and acting greedily with respect to the estimates with an arbitrary degree of entropy to ensure that the state-space is sufficiently explored. Behavior based on explicit greedification assumes that the values reflect those of textitsome policy, over which the greedy policy will be an improvement. However, value-iteration can produce value functions that do not correspond to textitany policy. This is especially relevant in the function-approximation regime, when the true value function can't be perfectly represented. In this work, we explore the use of textitinverse policy evaluation, the process of solving for a likely policy given a value function, for deriving behavior from a value function. We provide theoretical and empirical results to show that inverse policy evaluation, combined with an approximate value iteration algorithm, is a feasible method for value-based control.