mlsanityData to 14 Oct 2026

Performative Policy Gradient: Optimality in Performative Reinforcement Learning

arXiv 2512.20576 ↗First posted 23 Dec 2025cs.LG cs.AI math.OCcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Post-deployment machine learning algorithms often influence the environments they act in, and thus shift the underlying dynamics that the standard reinforcement learning (RL) methods ignore. While designing optimal algorithms in this performative setting has recently been studied in supervised learning, the RL counterpart remains under-explored. In this paper, we prove the performative counterparts of the performance difference lemma and the policy gradient theorem in RL, and further introduce the Performative Policy Gradient algorithm (PePG). PePG is the first policy gradient algorithm designed to account for performativity in RL. Under softmax parametrisation, and also with and without entropy regularisation, we prove that PePG converges to performatively optimal policies, i.e. policies that remain optimal under the distribution shifts induced by themselves. Thus, PePG significantly extends the prior works in Performative RL that achieves performative stability but not optimality. Furthermore, our empirical analysis on standard performative RL environments validate that PePG outperforms the existing performative RL algorithms aiming for stability.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 70 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

48 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025On Corruption-Robustness in Performative Reinforcement Learning8 May
  2. 2025Independent Learning in Performative Markov Potential Games29 Apr
  3. 2024Performative Reinforcement Learning with Linear Markov Decision Process7 Nov
  4. 2024Augmented Bayesian Policy Search5 Jul
  5. 2024Reward-Punishment Reinforcement Learning with Maximum Entropy20 May

Other references · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2020Performative Predictioncited by 327
  6. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  7. 2017A unified view of entropy-regularized Markov decision processescited by 182
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  9. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 120
  10. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  11. 2021Outside the Echo Chamber: Optimizing the Performative Riskcited by 96
  12. 2020Stochastic Optimization for Performative Predictioncited by 93
31 more
  1. 2020Feedback Loop and Bias Amplification in Recommender Systemscited by 90
  2. 2020Performative Prediction in a Stateful Worldcited by 90
  3. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  4. 2021How to Learn when Data Reacts to Your Model: Performative Gradient Descentcited by 71
  5. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  6. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 68
  7. 2022Multiplayer Performative Prediction: Learning in Decision-Dependent Gamescited by 44
  8. 2022Performative Powercited by 43
  9. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 40
  10. 2021State Dependent Performative Prediction with Stochastic Approximationcited by 39
  11. 2023Performative Prediction with Neural Networkscited by 34
  12. 2022Multi-agent Performative Prediction: From Global Stability and Optimality to Chaoscited by 33
  13. 2022Decision-Dependent Risk Minimization in Geometrically Decaying Dynamic Environmentscited by 32
  14. 2021How to Learn when Data Gradually Reacts to Your Modelcited by 30
  15. 2021Navigating to the Best Policy in Markov Decision Processescited by 29
  16. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  17. 2022Performative Reinforcement Learningcited by 27
  18. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  19. 2022Multi-agent Performative Prediction with Greedy Deployment and Consensus Seeking Agentscited by 23
  20. 2019Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learningcited by 20
  21. 2024Performative Reinforcement Learning in Gradually Shifting Environmentscited by 14
  22. 2019Optimal Best Markovian Arm Identification with Fixed Confidencecited by 12
  23. 2022Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration and Planningcited by 12
  24. 2023Performative Recommendation: Diversifying Content via Strategic Incentivescited by 12
  25. 2024Performative Prediction on Games and Mechanism Designcited by 7
  26. 2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6
  27. 2024Performative Control for Linear Dynamical Systemscited by 6
  28. 2025Decision-Dependent Stochastic Optimization: The Role of Distribution Dynamicscited by 6
  29. 2022Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertaintycited by 5
  30. 2023Optimal Best Arm Identification with Fixed Confidence in Restless Banditscited by 3
  31. 2024Practical Performative Policy Learning with Strategic Agentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 4
  2. 2026Active Inference as a Convex Markov Decision Processcited by 1
  3. 2026Simulating Eutopia: Revisiting Long-term Fairness with Outcomes, Performativity, and Dynamicscited by 0
  4. 2026Local and Global Stability in Performative Reinforcement Learningcited by 0
  5. 2026Self-Confirming Superposition Traps in Reinforcement Learningcited by 0
  6. 2026Partially Informed Stochastic Optimization via Zeroth-Order Residual Trackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Achieve Performatively Optimal Policy for Performative Reinforcement Learning
  2. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  3. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  4. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
  5. 2026Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies
  6. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  7. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  8. 2025Policy Gradient Guidance Enables Test Time Control