mlsanityData to 7 Oct 2026

An operator view of policy gradient methods

arXiv 2006.11266 ↗First posted 19 Jun 2020cs.LG cs.AI stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator I, which maps any policy π to a better one Iπ, and a projection operator P, which finds the best approximation of Iπ in the set of realizable policies. We use this framework to introduce operator-based versions of traditional policy gradient methods such as REINFORCE and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of I and P to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how REINFORCE and the Bellman optimality operator, for example, can be seen as two sides of the same coin.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 1; 2023: 3; 2024: 1; 2025: 4; 2026: 1.

17 papers.

Sort:

By area:

  1. 2021Learning and Planning in Complex Action Spacescited by 76
  2. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  3. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  4. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  5. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 16
  6. 2021Characterizing the Gap Between Actor-Critic and Policy Gradientcited by 9
  7. 2024Value Improved Actor Critic Algorithmscited by 6
  8. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  9. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  10. 2025The Hidden Link Between RLHF and Contrastive Learningcited by 5
  11. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuningcited by 4
  12. 2023Optimistic Multi-Agent Policy Gradientcited by 4
5 more
  1. 2021CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iterationcited by 1
  2. 2022A Parametric Class of Approximate Gradient Updates for Policy Optimizationcited by 0
  3. 2023Rule Enforcing Through Orderingcited by 0
  4. 2025Efficient Neural Clause-Selection Reinforcementcited by 0
  5. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policynesscited by 0