# Mean-Variance Optimization of Discrete Time Discounted Markov Decision Processes

arXiv: https://arxiv.org/abs/1708.07095 · HTML: https://mlsanity.com/paper/1708.07095/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2017.

## Built on by (most cited first, 2 of 2)

- [A unified algorithm framework for mean-variance optimization in discounted Markov decision processes](https://mlsanity.com/paper/2201.05737/) (2022)
- [Global Algorithms for Mean-Variance Optimization in Markov Decision Processes](https://mlsanity.com/paper/2302.13710/) (2023)

## Abstract (arXiv)

In this paper, we study a mean-variance optimization problem in an infinite horizon discrete time discounted Markov decision process (MDP). The objective is to minimize the variance of system rewards with the constraint of mean performance. Different from most of works in the literature which require the mean performance already achieve optimum, we can let the mean discounted performance equal any constant. The difficulty of this problem is caused by the quadratic form of the variance function which makes the variance minimization problem not a standard MDP. By proving the decomposable structure of the feasible policy space, we transform this constrained variance minimization problem to an equivalent unconstrained MDP under a new discounted criterion and a new reward function. The difference of the variances of Markov chains under any two feasible policies is quantified by a difference formula. Based on the variance difference formula, a policy iteration algorithm is developed to find the optimal policy. We also prove the optimality of deterministic policy over the randomized policy generated in the mean-constrained policy space. Numerical experiments demonstrate the effectiveness of our approach.