# Action-Constrained Markov Decision Processes With Kullback-Leibler Cost

arXiv: https://arxiv.org/abs/1807.10244 · HTML: https://mlsanity.com/paper/1807.10244/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2018.

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning Based Safe Decision Making for Highway Autonomous Driving](https://mlsanity.com/paper/2105.06517/) (2021)

## Abstract (arXiv)

This paper concerns computation of optimal policies in which the one-step reward function contains a cost term that models Kullback-Leibler divergence with respect to nominal dynamics. This technique was introduced by Todorov in 2007, where it was shown under general conditions that the solution to the average-reward optimality equations reduce to a simple eigenvector problem. Since then many authors have sought to apply this technique to control problems and models of bounded rationality in economics. A crucial assumption is that the input process is essentially unconstrained. For example, if the nominal dynamics include randomness from nature (e.g., the impact of wind on a moving vehicle), then the optimal control solution does not respect the exogenous nature of this disturbance. This paper introduces a technique to solve a more general class of action-constrained MDPs. The main idea is to solve an entire parameterized family of MDPs, in which the parameter is a scalar weighting the one-step reward function. The approach is new and practical even in the original unconstrained formulation.