mlsanityData to 7 Oct 2026

RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization

arXiv 2311.01753 ↗First posted 3 Nov 2023cs.MA cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

32 of 55 references.

References · 32

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  5. 2019The StarCraft Multi-Agent Challengecited by 589
  6. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  9. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  10. 2016Learning Multiagent Communication with Backpropagationcited by 355
  11. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  12. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
20 more
  1. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  2. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  3. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  4. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  5. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  6. 2014Algorithms for CVaR Optimization in MDPscited by 98
  7. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  8. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  9. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  10. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  11. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  12. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  13. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  14. 2020Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learningcited by 38
  15. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  16. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  17. 2021Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Compositioncited by 28
  18. 2021DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learningcited by 26
  19. 2019Learning Robust Options by Conditional Value at Risk Optimizationcited by 16
  20. 2022Risk Perspective Exploration in Distributional Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 8.

19 papers.

Sort:

By area:

  1. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  2. 2024On Stateful Value Factorization in Multi-Agent Reinforcement Learningcited by 5
  3. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 3
  4. 2026Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learningcited by 3
  5. 2025Trajectory-Class-Aware Multi-Agent Reinforcement Learningcited by 2
  6. 2026Training Generalizable Collaborative Agents via Strategic Risk Aversioncited by 2
  7. 2025Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamicscited by 1
  8. 2025Enhancing Sample Efficiency in Multi-Agent RL with Uncertainty Quantification and Selective Explorationcited by 1
  9. 2024Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observationcited by 0
  10. 2024Risk Sensitivity in Markov Games and Multi-Agent Reinforcement Learning: A Systematic Reviewcited by 0
  11. 2025Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependenciescited by 0
  12. 2025Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Gamescited by 0
7 more
  1. 2025Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systemscited by 0
  2. 2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0
  3. 2026Strategically Robust Multi-Agent Reinforcement Learning with Linear Function Approximationcited by 0
  4. 2026Market Regime Council for Dynamic Credit Assignment in Multi-Agent LLM Decision Systemscited by 0
  5. 2026MASK: Multi-Agent Semantic K-Scheduling for Risk-Sensitive 6G Roboticscited by 0
  6. 2026Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methodscited by 0
  7. 2026AIGP: An LLM-Based Framework for Long-Term Value Alignment in E-Commerce Pricingcited by 0