# Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.08630 · HTML: https://mlsanity.com/paper/2505.08630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 27 of 27)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07489/) (2022)
- [Off-Policy Multi-Agent Decomposed Policy Gradients](https://mlsanity.com/paper/2007.12322/) (2020)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [Influence-Based Multi-Agent Exploration](https://mlsanity.com/paper/1910.05512/) (2019)
- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer](https://mlsanity.com/paper/2206.10607/) (2022)
- [CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1809.05188/) (2018)
- [PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration](https://mlsanity.com/paper/2203.08553/) (2022)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Changing the Environment Based on Empowerment as Intrinsic Motivation](https://mlsanity.com/paper/1406.1767/) (2014)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [FoX: Formation-aware exploration in multi-agent reinforcement learning](https://mlsanity.com/paper/2308.11272/) (2023)
- [Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.17540/) (2022)
- [Multi-Target Pursuit by a Decentralized Heterogeneous UAV Swarm using Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.01799/) (2023)
- [An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards](https://mlsanity.com/paper/2010.07986/) (2020)

## Abstract (arXiv)

Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods struggle with precise credit assignment among agents and effective exploration. In this paper, we introduce a novel method to deal with both credit assignment and exploration problems in reward-sparse domains. Accordingly, we propose an algorithm that calculates the Influence Scope of Agents (ISA) on states by taking specific value of the dimensions/attributes of states that can be influenced by individual agents. The mutual dependence between agents' actions and state attributes are then used to calculate the credit assignment and to delimit the exploration space for each individual agent. We then evaluate ISA in a variety of sparse-reward multi-agent scenarios. The results show that our method significantly outperforms the state-of-art baselines.