# A nearly Blackwell-optimal policy gradient method

arXiv: https://arxiv.org/abs/2105.13609 · HTML: https://mlsanity.com/paper/2105.13609/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 35 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 9 of 9)

- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies](https://mlsanity.com/paper/1906.08383/) (2019)
- [Policy Evaluation Networks](https://mlsanity.com/paper/2002.11833/) (2020)
- [Parameter-Based Value Functions](https://mlsanity.com/paper/2006.09226/) (2020)
- [Sample Complexity of Policy Gradient Finding Second-Order Stationary Points](https://mlsanity.com/paper/2012.01491/) (2020)
- [Zeroth-order Deterministic Policy Gradient](https://mlsanity.com/paper/2006.07314/) (2020)
- [Fast Efficient Hyperparameter Tuning for Policy Gradients](https://mlsanity.com/paper/1902.06583/) (2019)
- [Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications](https://mlsanity.com/paper/2004.00857/) (2020)
- [Issues concerning realizability of Blackwell optimal policies in reinforcement learning](https://mlsanity.com/paper/1905.08293/) (2019)

## Built on by (most cited first, 1 of 1)

- [Efficient Computation of Blackwell Optimal Policies using Rational Functions](https://mlsanity.com/paper/2508.18252/) (2025)

## Abstract (arXiv)

For continuing environments, reinforcement learning (RL) methods commonly maximize the discounted reward criterion with discount factor close to 1 in order to approximate the average reward (the gain). However, such a criterion only considers the long-run steady-state performance, ignoring the transient behaviour in transient states. In this work, we develop a policy gradient method that optimizes the gain, then the bias (which indicates the transient performance and is important to capably select from policies with equal gain). We derive expressions that enable sampling for the gradient of the bias and its preconditioning Fisher matrix. We further devise an algorithm that solves the gain-then-bias (bi-level) optimization. Its key ingredient is an RL-specific logarithmic barrier function. Experimental results provide insights into the fundamental mechanisms of our proposal.