# A Game-Theoretic Framework for Managing Risk in Multi-Agent Systems

arXiv: https://arxiv.org/abs/2205.15434 · HTML: https://mlsanity.com/paper/2205.15434/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 69 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 14 of 14)

- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Risk-Averse Trust Region Optimization for Reward-Volatility Reduction](https://mlsanity.com/paper/1912.03193/) (2019)
- [Prosocial learning agents solve generalized Stag Hunts better than selfish ones](https://mlsanity.com/paper/1709.02865/) (2017)
- [Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games](https://mlsanity.com/paper/2006.08555/) (2020)
- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [XDO: A Double Oracle Algorithm for Extensive-Form Games](https://mlsanity.com/paper/2103.06426/) (2021)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2004.10888/) (2020)
- [Online Double Oracle](https://mlsanity.com/paper/2103.07780/) (2021)
- [Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty](https://mlsanity.com/paper/2203.10045/) (2022)
- [Risk-Averse Equilibrium for Games](https://mlsanity.com/paper/2002.08414/) (2020)
- [Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets](https://mlsanity.com/paper/2107.08083/) (2021)
- [Congestion Games with Distance-Based Strict Uncertainty](https://mlsanity.com/paper/1411.4943/) (2014)
- [Sample average approximation of CVaR-based Wardrop equilibrium in routing under uncertain costs](https://mlsanity.com/paper/1909.03783/) (2019)

## Built on by (most cited first, 14 of 14)

- [Policy Space Response Oracles: A Survey](https://mlsanity.com/paper/2403.02227/) (2024)
- [Tractable Equilibrium Computation in Markov Games through Risk Aversion](https://mlsanity.com/paper/2406.14156/) (2024)
- [Curiosity Meets Cooperation: A Game-Theoretic Approach to Long-Tail Multi-Label Learning](https://mlsanity.com/paper/2510.17520/) (2025)
- [Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs](https://mlsanity.com/paper/2506.00577/) (2025)
- [Training Generalizable Collaborative Agents via Strategic Risk Aversion](https://mlsanity.com/paper/2602.21515/) (2026)
- [Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment](https://mlsanity.com/paper/2604.05965/) (2026)
- [Strategic Integration of Artificial Intelligence in the C-Suite: The Role of the Chief AI Officer](https://mlsanity.com/paper/2407.10247/) (2024)
- [Cyber Shadows: Neutralizing Security Threats with AI and Targeted Policy Measures](https://mlsanity.com/paper/2501.09025/) (2025)
- [Position: Emergent Machina Sapiens Urge Rethinking Multi-Agent Paradigms](https://mlsanity.com/paper/2502.04388/) (2025)
- [Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Games](https://mlsanity.com/paper/2503.22779/) (2025)
- [Stochastic Approximation Methods for Distortion Risk Measure Optimization](https://mlsanity.com/paper/2510.04563/) (2025)
- [Strategically Robust Multi-Agent Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2603.09208/) (2026)
- [Beyond Bayesian Nash: Learning Minimax-Regret Equilibria for Adversarial Team Games under Asymmetric Information](https://mlsanity.com/paper/2607.09993/) (2026)
- [Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident](https://mlsanity.com/paper/2610.00902/) (2026)

## Abstract (arXiv)

In order for agents in multi-agent systems (MAS) to be safe, they need to take into account the risks posed by the actions of other agents. However, the dominant paradigm in game theory (GT) assumes that agents are not affected by risk from other agents and only strive to maximise their expected utility. For example, in hybrid human-AI driving systems, it is necessary to limit large deviations in reward resulting from car crashes. Although there are equilibrium concepts in game theory that take into account risk aversion, they either assume that agents are risk-neutral with respect to the uncertainty caused by the actions of other agents, or they are not guaranteed to exist. We introduce a new GT-based Risk-Averse Equilibrium (RAE) that always produces a solution that minimises the potential variance in reward accounting for the strategy of other agents. Theoretically and empirically, we show RAE shares many properties with a Nash Equilibrium (NE), establishing convergence properties and generalising to risk-dominant NE in certain cases. To tackle large-scale problems, we extend RAE to the PSRO multi-agent reinforcement learning (MARL) framework. We empirically demonstrate the minimum reward variance benefits of RAE in matrix games with high-risk outcomes. Results on MARL experiments show RAE generalises to risk-dominant NE in a trust dilemma game and that it reduces instances of crashing by 7x in an autonomous driving setting versus the best performing baseline.