# Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games

arXiv: https://arxiv.org/abs/2402.05906 · HTML: https://mlsanity.com/paper/2402.05906/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 66 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games](https://mlsanity.com/paper/2106.01969/) (2021)
- [Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity](https://mlsanity.com/paper/2007.07461/) (2020)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Risk-Averse Offline Reinforcement Learning](https://mlsanity.com/paper/2102.05371/) (2021)
- [Decentralized Q-Learning in Zero-sum Markov Games](https://mlsanity.com/paper/2106.02748/) (2021)
- [Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence](https://mlsanity.com/paper/2202.04129/) (2022)
- [Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.03947/) (2021)
- [Independent Natural Policy Gradient Always Converges in Markov Potential Games](https://mlsanity.com/paper/2110.10614/) (2021)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPs](https://mlsanity.com/paper/1403.6530/) (2014)
- [Learning in Nonzero-Sum Stochastic Games with Potentials](https://mlsanity.com/paper/2103.09284/) (2021)
- [Independent and Decentralized Learning in Markov Potential Games](https://mlsanity.com/paper/2205.14590/) (2022)
- [A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP](https://mlsanity.com/paper/2202.04157/) (2022)
- [Predicting Human Behavior in Unrepeated, Simultaneous-Move Games](https://mlsanity.com/paper/1306.0918/) (2013)
- [Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitions](https://mlsanity.com/paper/2207.12463/) (2022)
- [Leader-Follower Network Aggregative Game with Stochastic Agents' Communication and Activeness](https://mlsanity.com/paper/1908.02458/) (2019)
- [Risk-Aware Energy Scheduling for Edge Computing with Microgrid: A Multi-Agent Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2003.02157/) (2020)
- [Ultra-Reliable Indoor Millimeter Wave Communications using Multiple Artificial Intelligence-Powered Intelligent Surfaces](https://mlsanity.com/paper/2104.00075/) (2021)
- [Nonzero-sum risk-sensitive continuous-time stochastic games with ergodic costs](https://mlsanity.com/paper/2106.00255/) (2021)
- [Discrete-time Zero-Sum Games for Markov chains with risk-sensitive average cost criterion](https://mlsanity.com/paper/2201.03790/) (2022)

## Built on by (most cited first, 1 of 1)

- [Risk Sensitivity in Markov Games and Multi-Agent Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2406.06041/) (2024)

## Abstract (arXiv)

Classical multi-agent reinforcement learning (MARL) assumes risk neutrality and complete objectivity for agents. However, in settings where agents need to consider or model human economic or social preferences, a notion of risk must be incorporated into the RL optimization problem. This will be of greater importance in MARL where other human or non-human agents are involved, possibly with their own risk-sensitive policies. In this work, we consider risk-sensitive and non-cooperative MARL with cumulative prospect theory (CPT), a non-convex risk measure and a generalization of coherent measures of risk. CPT is capable of explaining loss aversion in humans and their tendency to overestimate/underestimate small/large probabilities. We propose a distributed sampling-based actor-critic (AC) algorithm with CPT risk for network aggregative Markov games (NAMGs), which we call Distributed Nested CPT-AC. Under a set of assumptions, we prove the convergence of the algorithm to a subjective notion of Markov perfect Nash equilibrium in NAMGs. The experimental results show that subjective CPT policies obtained by our algorithm can be different from the risk-neutral ones, and agents with a higher loss aversion are more inclined to socially isolate themselves in an NAMG.