# Refined Policy Improvement Bounds for MDPs

arXiv: https://arxiv.org/abs/2107.08068 · HTML: https://mlsanity.com/paper/2107.08068/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 16 Jul 2021.

## Builds on (most cited first, 7 of 7)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Queueing Network Controls via Deep Reinforcement Learning](https://mlsanity.com/paper/2008.01644/) (2020)
- [On-Policy Deep Reinforcement Learning for the Average-Reward Criterion](https://mlsanity.com/paper/2106.07329/) (2021)

## Built on by (most cited first, 3 of 3)

- [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023)
- [Enhancing PPO with Trajectory-Aware Hybrid Policies](https://mlsanity.com/paper/2502.15968/) (2025)
- [Processing Network Controls via Deep Reinforcement Learning](https://mlsanity.com/paper/2205.02119/) (2022)

## Abstract (arXiv)

The policy improvement bound on the difference of the discounted returns plays a crucial role in the theoretical justification of the trust-region policy optimization (TRPO) algorithm. The existing bound leads to a degenerate bound when the discount factor approaches one, making the applicability of TRPO and related algorithms questionable when the discount factor is close to one. We refine the results in citeSchulman2015, Achiam2017 and propose a novel bound that is "continuous" in the discount factor. In particular, our bound is applicable for MDPs with the long-run average rewards as well.