mlsanityData to 7 Oct 2026

Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games

arXiv 2107.13090 ↗First posted 27 Jul 2021math.OC cs.GT cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We consider a general-sum N-player linear-quadratic game with stochastic dynamics over a finite horizon and prove the global convergence of the natural policy gradient method to the Nash equilibrium. In order to prove the convergence of the method, we require a certain amount of noise in the system. We give a condition, essentially a lower bound on the covariance of the noise in terms of the model parameters, in order to guarantee convergence. We illustrate our results with numerical experiments to show that even in situations where the policy gradient method may not converge in the deterministic setting, the addition of noise leads to convergence.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 22 of its 38 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

22 of 38 references.

References · 22

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  3. 2017Learning with Opponent-Learning Awarenesscited by 218
  4. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
  5. 2017Cycles in adversarial regularized learningcited by 122
  6. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  7. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
  8. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65
  9. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  10. 2020No-regret learning and mixed Nash equilibria: They do not mixcited by 49
  11. 2018On Gradient-Based Learning in Continuous Gamescited by 46
  12. 2019Differentiable Game Mechanicscited by 43
10 more
  1. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 43
  2. 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 38
  3. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 31
  4. 2013Nash Convergence of Gradient Dynamics in Iterated General-Sum Gamescited by 18
  5. 2018Global Convergence to the Equilibrium of GANs using Variational Inequalitiescited by 15
  6. 2019Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Gamescited by 15
  7. 2021Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexitycited by 15
  8. 2019Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Gamescited by 14
  9. 2019Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Gamescited by 8
  10. 2020Reinforcement Learning in Nonzero-sum Linear Quadratic Deep Structured Games: Global Convergence of Policy Optimizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 4; 2024: 3; 2025: 2; 2026: 3.

19 papers.

Sort:

By area:

  1. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
  2. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 19
  3. 2023Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Gamescited by 12
  4. 2022Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulationscited by 10
  5. 2023Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methodscited by 9
  6. 2022Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergencecited by 8
  7. 2021Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methodscited by 7
  8. 2022Learning Stationary Nash Equilibrium Policies in n-Player Stochastic Games with Independent Chainscited by 6
  9. 2023Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity and Last-Iterate Convergencecited by 3
  10. 2026Model-free policy gradient for discrete-time mean-field controlcited by 3
  11. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspectivecited by 2
  12. 2024Policy Optimization finds Nash Equilibrium in Regularized General-Sum LQ Gamescited by 2
7 more
  1. 2023Scalable and Independent Learning of Nash Equilibrium Policies in n-Player Stochastic Games with Unknown Independent Chainscited by 1
  2. 2024TrafficGamer: Reliable and Flexible Traffic Simulation for Safety-Critical Scenarios with Game-Theoretic Oraclescited by 1
  3. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Gamescited by 1
  4. 2026Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An α-Potential Approachcited by 1
  5. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  6. 2025Online Multi-Agent Control with Adversarial Disturbancescited by 0
  7. 2026Learning under Opponent Unawareness in Linear-Quadratic Stochastic Gamescited by 0