mlsanityData to 14 Oct 2026

Convergent Q-Learning for Infinite-Horizon General-Sum Markov Games through Behavioral Economics

arXiv 2508.08669 ↗First posted 12 Aug 2025cs.GTcited by 2

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Risk-aversion and bounded rationality are two key characteristics of human decision-making. Risk-averse quantal-response equilibrium (RQE) is a solution concept that incorporates these features, providing a more realistic depiction of human decision making in various strategic environments compared to a Nash equilibrium. Furthermore a class of RQE has recently been shown in arXiv:2406.14156 to be universally computationally tractable in all finite-horizon Markov games, allowing for the development of multi-agent reinforcement learning algorithms with convergence guarantees. In this paper, we expand upon the study of RQE and analyze their computation in both two-player normal form games and discounted infinite-horizon Markov games. For normal form games we adopt a monotonicity-based approach allowing us to generalize previous results. We first show uniqueness and Lipschitz continuity of RQE with respect to player's payoff matrices under monotonicity assumptions, and then provide conditions on the players' degrees of risk aversion and bounded rationality that ensure monotonicity. We then focus on discounted infinite-horizon Markov games. We define the risk-averse quantal-response Bellman operator and prove its contraction under further conditions on the players' risk-aversion, bounded rationality, and temporal discounting. This yields a Q-learning based algorithm with convergence guarantees for all infinite-horizon general-sum Markov games.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 21 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

9 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversion20 Jun

Other references · 8

  1. 2017Cycles in adversarial regularized learningcited by 124
  2. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  3. 2020Tight last-iterate convergence rates for no-regret learning in multi-player gamescited by 49
  4. 2022The Complexity of Markov Equilibrium in Stochastic Gamescited by 49
  5. 2018On Gradient-Based Learning in Continuous Gamescited by 48
  6. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
  7. 2023Doubly Optimal No-Regret Learning in Monotone Gamescited by 14
  8. 2021Bounded rationality for relaxing best response and mutual consistency: The Quantal Hierarchy model of decision-makingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Training Generalizable Collaborative Agents via Strategic Risk Aversioncited by 2
  2. 2026Strategically Robust Multi-Agent Reinforcement Learning with Linear Function Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Provably Convergent Actor-Critic for MARL through Risk-aversion
  2. 2026Robust Mean-Field Games with Risk Aversion and Bounded Rationality
  3. 2024On the Stability of Learning in Network Games with Many Players
  4. 2024Asymptotic Extinction in Large Coordination Games
  5. 2024Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework
  6. 2025Risk-Averse Total-Reward Reinforcement Learning
  7. 2024widetildeO(T-1) Convergence to (Coarse) Correlated Equilibria in Full-Information General-Sum Markov Games
  8. 2024Risk Sensitivity in Markov Games and Multi-Agent Reinforcement Learning: A Systematic Review