Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
In multi-agent environments, agents often struggle to learn optimal policies due to sparse or delayed global rewards, particularly in long-horizon tasks where it is challenging to evaluate actions at intermediate time steps. We introduce Temporal-Agent Reward Redistribution (TAR2), a novel approach designed to address the agent-temporal credit assignment problem by redistributing sparse rewards both temporally and across agents. TAR2 decomposes sparse global rewards into time-step-specific rewards and calculates agent-specific contributions to these rewards. We theoretically prove that TAR2 is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirical results demonstrate that TAR2 stabilizes and accelerates the learning process. Additionally, we show that when TAR2 is integrated with single-agent reinforcement learning algorithms, it performs as well as or better than traditional multi-agent reinforcement learning methods.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 19 Dec 2024.
Builds on
26 of 37 references.
References · 26
- 2017Attention Is All You Needcited by 59,932
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
- 2017Counterfactual Multi-Agent Policy Gradientscited by 733
- 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
- 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
- 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
- 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
- 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
14 more
- 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
- 2019Hindsight Credit Assignmentcited by 74
- 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 71
- 2020Reinforcement Learning with Trajectory Feedbackcited by 40
- 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 35
- 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
- 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
- 2019Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learningcited by 27
- 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
- 2022Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learningcited by 9
- 2014Policy Invariance under Reward Transformations for General-Sum Stochastic Gamescited by 6
- 2022Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learningcited by 6
- 2023SMAClite: A Lightweight Environment for Multi-Agent Reinforcement Learningcited by 6
- 2021Learning Cooperative Multi-Agent Policies with Partial Reward Decouplingcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.