# Revisit Policy Optimization in Matrix Form

arXiv: https://arxiv.org/abs/1909.09186 · HTML: https://mlsanity.com/paper/1909.09186/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 2 of 2)

- [On Addressing the Limitations of Graph Neural Networks](https://mlsanity.com/paper/2306.12640/) (2023)
- [Adaptive Policy Backbone via Shared Network](https://mlsanity.com/paper/2509.22310/) (2025)

## Abstract (arXiv)

In tabular case, when the reward and environment dynamics are known, policy evaluation can be written as bmV_bmπ = (I - γ P_bmπ)-1 bmr_bmπ, where P_bmπ is the state transition matrix given policy bmπ and bmr_bmπ is the reward signal given bmπ. What annoys us is that P_bmπ and bmr_bmπ are both mixed with bmπ, which means every time when we update bmπ, they will change together. In this paper, we leverage the notation from citewang2007dual to disentangle bmπ and environment dynamics which makes optimization over policy more straightforward. We show that policy gradient theorem citesutton2018reinforcement and TRPO citeschulman2015trust can be put into a more general framework and such notation has good potential to be extended to model-based reinforcement learning.