mlsanityData to 14 Oct 2026

Independent Learning in Performative Markov Potential Games

arXiv 2504.20593 ↗First posted 29 Apr 2025cs.LGcited by 5

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Performative Reinforcement Learning (PRL) refers to a scenario in which the deployed policy changes the reward and transition dynamics of the underlying environment. In this work, we study multi-agent PRL by incorporating performative effects into Markov Potential Games (MPGs). We introduce the notion of a performatively stable equilibrium (PSE) and show that it always exists under a reasonable sensitivity assumption. We then provide convergence results for state-of-the-art algorithms used to solve MPGs. Specifically, we show that independent policy gradient ascent (IPGA) and independent natural policy gradient (INPG) converge to an approximate PSE in the best-iterate sense, with an additional term that accounts for the performative effects. Furthermore, we show that INPG asymptotically converges to a PSE in the last-iterate sense. As the performative effects vanish, we recover the convergence rates from prior work. For a special case of our game, we provide finite-time last-iterate convergence results for a repeated retraining approach, in which agents independently optimize a surrogate objective. We conduct extensive experiments to validate our theoretical findings.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 51 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

30 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Performative Reinforcement Learning with Linear Markov Decision Process7 Nov

Other references · 29

  1. 2020Performative Predictioncited by 327
  2. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 93
  3. 2020Performative Prediction in a Stateful Worldcited by 90
  4. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 57
  5. 2022Multiplayer Performative Prediction: Learning in Decision-Dependent Gamescited by 44
  6. 2022Performative Powercited by 43
  7. 2022Regret Minimization with Performative Feedbackcited by 40
  8. 2021State Dependent Performative Prediction with Stochastic Approximationcited by 39
  9. 2021Independent Natural Policy Gradient Always Converges in Markov Potential Gamescited by 37
  10. 2022Anticipating Performativity by Predicting from Predictionscited by 37
  11. 2021On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learningcited by 36
  12. 2023Performative Prediction with Neural Networkscited by 34
17 more
  1. 2022Multi-agent Performative Prediction: From Global Stability and Optimality to Chaoscited by 33
  2. 2022Decision-Dependent Risk Minimization in Geometrically Decaying Dynamic Environmentscited by 32
  3. 2021How to Learn when Data Gradually Reacts to Your Modelcited by 30
  4. 2022Performative Reinforcement Learningcited by 27
  5. 2022Multi-agent Performative Prediction with Greedy Deployment and Consensus Seeking Agentscited by 23
  6. 2022On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Gamescited by 22
  7. 2022Independent and Decentralized Learning in Markov Potential Gamescited by 22
  8. 2018Multi-agent online learning in time-varying gamescited by 17
  9. 2023On the Convergence of No-Regret Learning Dynamics in Time-Varying Gamescited by 14
  10. 2024Performative Reinforcement Learning in Gradually Shifting Environmentscited by 14
  11. 2023Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Gamescited by 13
  12. 2024Independent Learning in Constrained Markov Potential Gamescited by 10
  13. 2023Convergence Rates for Localized Actor-Critic in Networked Markov Potential Gamescited by 9
  14. 2023Provably Learning Nash Policies in Constrained Markov Potential Gamescited by 7
  15. 2024Independent Policy Mirror Descent for Markov Potential Games: Scaling to Large Number of Playerscited by 6
  16. 2024Performative Control for Linear Dynamical Systemscited by 6
  17. 2023A Black-box Approach for Non-stationary Multi-agent Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Performative Policy Gradient: Optimality in Performative Reinforcement Learningcited by 6
  2. 2025Achieve Performatively Optimal Policy for Performative Reinforcement Learningcited by 2
  3. 2026Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibriacited by 1
  4. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learningcited by 1
  5. 2026Local and Global Stability in Performative Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  2. 2024Linear Convergence of Independent Natural Policy Gradient in Games with Entropy Regularization
  3. 2026Internal State-Based Policy Gradient Methods for Partially Observable Markov Potential Games
  4. 2024Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework
  5. 2026Provably Convergent Actor-Critic for MARL through Risk-aversion
  6. 2024Almost Sure Convergence of Networked Policy Gradient over Time-Varying Networks in Markov Potential Games
  7. 2026Independent Learning of Nash Equilibria in Partially Observable Markov Potential Games with Decoupled Dynamics
  8. 2024Provable Policy Gradient Methods for Average-Reward Markov Potential Games