Revisit Policy Optimization in Matrix Form

In tabular case, when the reward and environment dynamics are known, policy evaluation can be written as bmV_bmπ = (I - γ P_bmπ)-1 bmr_bmπ, where P_bmπ is the state transition matrix given policy bmπ and bmr_bmπ is the reward signal given bmπ. What annoys us is that P_bmπ and bmr_bmπ are both mixed with bmπ, which means every time when we update bmπ, they will change together. In this paper, we leverage the notation from citewang2007dual to disentangle bmπ and environment dynamics which makes optimization over policy more straightforward. We show that policy gradient theorem citesutton2018reinforcement and TRPO citeschulman2015trust can be put into a more general framework and such notation has good potential to be extended to model-based reinforcement learning.