mlsanityData to 14 Oct 2026

Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning

arXiv 2508.15652 ↗First posted 21 Aug 2025cs.AI cs.IT cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

To reliably deploy Multi-Agent Reinforcement Learning (MARL) systems, it is crucial to understand individual agent behaviors. While prior work typically evaluates overall team performance based on explicit reward signals, it is unclear how to infer agent contributions in the absence of any value feedback. In this work, we investigate whether meaningful insights into agent behaviors can be extracted solely by analyzing the policy distribution. Inspired by the phenomenon that intelligent agents tend to pursue convergent instrumental values, we introduce Intended Cooperation Values (ICVs), a method based on information-theoretic Shapley values for quantifying each agent's causal influence on their co-players' instrumental empowerment. Specifically, ICVs measure an agent's action effect on its teammates' policies by assessing their decision (un)certainty and preference alignment. By analyzing action effects on policies and value functions across cooperative and competitive MARL tasks, our method identifies which agent behaviors are beneficial to team success, either by fostering deterministic decisions or by preserving flexibility for future action choices, while also revealing the extent to which agents adopt similar or diverse strategies. Our proposed method offers novel insights into cooperation dynamics and enhances explainability in MARL systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

28 of 46 references.

References · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2019The many Shapley values for model explanationcited by 344
  4. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  5. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  6. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  7. 2019Optimal Policies Tend to Seek Powercited by 119
  8. 2019Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainabilitycited by 114
  9. 2020Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Modelscited by 107
  10. 2020On the Tractability of SHAP Explanationscited by 92
  11. 2020Shapley explainability on the data manifoldcited by 88
  12. 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 73
16 more
  1. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 50
  2. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  3. 2023Explaining Predictive Uncertainty with Information Theoretic Shapley Valuescited by 36
  4. 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methodscited by 31
  5. 2023A Review of Cooperation in Multi-agent Learningcited by 29
  6. 2021SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learningcited by 26
  7. 2023Explaining Reinforcement Learning with Shapley Valuescited by 26
  8. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  9. 2023TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Playcited by 20
  10. 2019A Unified Bellman Optimality Principle Combining Reward Maximization and Empowermentcited by 15
  11. 2023NA2Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learningcited by 14
  12. 2023Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learningcited by 10
  13. 2021Learning Altruistic Behaviours in Reinforcement Learning without External Rewardscited by 8
  14. 2022Trade-off Between Efficiency and Consistency for Removal-based Explanationscited by 6
  15. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Taskscited by 5
  16. 2023Efficiently Quantifying Individual Agent Importance in Cooperative MARLcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
  2. 2024Innate-Values-driven Reinforcement Learning based Cooperative Multi-Agent Cognitive Modeling
  3. 2026Coordination Matters: Evaluation of Cooperative Multi-Agent Reinforcement Learning
  4. 2024The impact of behavioral diversity in multi-agent reinforcement learning
  5. 2025An Organizationally-Oriented Approach to Enhancing Explainability and Control in Multi-Agent Reinforcement Learning
  6. 2025A Theoretical Framework for Explaining Reinforcement Learning with Shapley Values
  7. 2025MACIE: Multi-Agent Causal Intelligence Explainer for Collective Behavior Understanding
  8. 2026Action-Graph Policies: Learning Action Co-dependencies in Multi-Agent Reinforcement Learning