# Guided Policy Exploration for Markov Decision Processes using an Uncertainty-Based Value-of-Information Criterion

arXiv: https://arxiv.org/abs/1802.01518 · HTML: https://mlsanity.com/paper/1802.01518/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2018.

## Built on by (most cited first, 3 of 3)

- [The Conditional Cauchy-Schwarz Divergence with Applications to Time-Series Data and Sequential Decision Making](https://mlsanity.com/paper/2301.08970/) (2023)
- [Annotating Motion Primitives for Simplifying Action Search in Reinforcement Learning](https://mlsanity.com/paper/2102.12017/) (2021)
- [An Information-Theoretic Approach for Automatically Determining the Number of States when Aggregating Markov Chains](https://mlsanity.com/paper/2107.01799/) (2021)

## Abstract (arXiv)

Reinforcement learning in environments with many action-state pairs is challenging. At issue is the number of episodes needed to thoroughly search the policy space. Most conventional heuristics address this search problem in a stochastic manner. This can leave large portions of the policy space unvisited during the early training stages. In this paper, we propose an uncertainty-based, information-theoretic approach for performing guided stochastic searches that more effectively cover the policy space. Our approach is based on the value of information, a criterion that provides the optimal trade-off between expected costs and the granularity of the search process. The value of information yields a stochastic routine for choosing actions during learning that can explore the policy space in a coarse to fine manner. We augment this criterion with a state-transition uncertainty factor, which guides the search process into previously unexplored regions of the policy space.