mlsanityData to 7 Oct 2026

Performative Reinforcement Learning

arXiv 2207.00046 ↗First posted 30 Jun 2022cs.LG cs.GTcited by 25

Not on the map. Why?

Abstract (arXiv)

We introduce the framework of performative reinforcement learning where the policy chosen by the learner affects the underlying reward and transition dynamics of the environment. Following the recent literature on performative prediction~citePerdomo et. al., 2020, we introduce the concept of performatively stable policy. We then consider a regularized version of the reinforcement learning problem and show that repeatedly optimizing this objective converges to a performatively stable policy under reasonable assumptions on the transition dynamics. Our proof utilizes the dual perspective of the reinforcement learning problem and may be of independent interest in analyzing the convergence of other algorithms with decision-dependent environments. We then extend our results for the setting where the learner just performs gradient ascent steps instead of fully optimizing the objective, and for the setting where the learner has access to a finite number of trajectories from the changed environment. For both settings, we leverage the dual formulation of performative reinforcement learning and establish convergence to a stable solution. Finally, through extensive experiments on a grid-world environment, we demonstrate the dependence of convergence on various parameters e.g. regularization, smoothness, and the number of samples.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

34 of 61 references.

References · 34

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  4. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  5. 2020Performative Predictioncited by 318
  6. 2017A unified view of entropy-regularized Markov decision processescited by 174
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  8. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  9. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 116
  10. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  11. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  12. 2021Outside the Echo Chamber: Optimizing the Performative Riskcited by 95
22 more
  1. 2020Stochastic Optimization for Performative Predictioncited by 92
  2. 2020Performative Prediction in a Stateful Worldcited by 90
  3. 2020Feedback Loop and Bias Amplification in Recommender Systemscited by 89
  4. 2019Online Convex Optimization in Adversarial Markov Decision Processescited by 79
  5. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 75
  6. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  7. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  8. 2022Multiplayer Performative Prediction: Learning in Decision-Dependent Gamescited by 42
  9. 2021Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Gamescited by 40
  10. 2017Cooperating with Machinescited by 39
  11. 2018Fighting Fire with Fire: Using Antidote Data to Improve Polarization and Fairness of Recommender Systemscited by 37
  12. 2020Dynamic Regret of Policy Optimization in Non-stationary Environmentscited by 32
  13. 2020A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescited by 32
  14. 2018A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitionscited by 29
  15. 2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 26
  16. 2021Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?cited by 24
  17. 2017Game-Theoretic Modeling of Human Adaptation in Human-Robot Collaborationcited by 21
  18. 2020Learning to Play Sequential Games versus Unknown Opponentscited by 18
  19. 2019Learning to Collaborate in Markov Decision Processescited by 14
  20. 2021On Blame Attribution for Accountable Multi-Agent Sequential Decision Makingcited by 13
  21. 2020Model-free Reinforcement Learning for Stochastic Stackelberg Security Gamescited by 4
  22. 2015Learning Adversary Behavior in Security Games: A PAC Model Perspectivecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 6; 2025: 7; 2026: 8.

25 papers.

Sort:

By area:

  1. 2023Performative Prediction: Past and Futurecited by 52
  2. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  3. 2024Performative Reinforcement Learning in Gradually Shifting Environmentscited by 13
  4. 2023Zero-Regret Performative Prediction Under Inequality Constraintscited by 10
  5. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  6. 2023Performative Time-Series Forecastingcited by 8
  7. 2024Performative Reinforcement Learning with Linear Markov Decision Processcited by 8
  8. 2026Self-Evolving Agents with Anytime-Valid Certificatescited by 6
  9. 2025Decision-Dependent Stochastic Optimization: The Role of Distribution Dynamicscited by 5
  10. 2025Independent Learning in Performative Markov Potential Gamescited by 5
  11. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4
  12. 2025Performative Policy Gradient: Optimality in Performative Reinforcement Learningcited by 4
13 more
  1. 2022Semi-supervised Batch Learning From Logged Datacited by 2
  2. 2025Nonlinear Performative Predictioncited by 2
  3. 2025Achieve Performatively Optimal Policy for Performative Reinforcement Learningcited by 2
  4. 2026Dissecting Performative Prediction: A Comprehensive Surveycited by 2
  5. 2026Active Inference as a Convex Markov Decision Processcited by 1
  6. 2024Adaptive Discounting of Training Time Attackscited by 0
  7. 2024Reciprocal Learningcited by 0
  8. 2025Towards Performatively Stable Equilibria in Decision-Dependent Games for Arbitrary Data Distribution Mapscited by 0
  9. 2026When Predictions Shape Reality: A Socio-Technical Synthesis of Performative Predictions in Machine Learningcited by 0
  10. 2026Simulating Eutopia: Revisiting Long-term Fairness with Outcomes, Performativity, and Dynamicscited by 0
  11. 2026A Fair Objective for Human-Empowerment-Preserving AI: Desiderata, Design, and Likely Behavioral Consequencescited by 0
  12. 2026Local and Global Stability in Performative Reinforcement Learningcited by 0
  13. 2026When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worldscited by 0