# Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.17540 · HTML: https://mlsanity.com/paper/2210.17540/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [A Review of Cooperative Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03963/) (2019)
- [Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning](https://mlsanity.com/paper/1510.08906/) (2015)
- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2006.11438/) (2020)
- [Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1911.04175/) (2019)
- [Deep Reinforcement Learning for Swarm Systems](https://mlsanity.com/paper/1807.06613/) (2018)
- [Multi-Agent Collaboration via Reward Attribution Decomposition](https://mlsanity.com/paper/2010.08531/) (2020)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [Coordinated Reinforcement Learning for Optimizing Mobile Networks](https://mlsanity.com/paper/2109.15175/) (2021)

## Built on by (most cited first, 6 of 6)

- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.08936/) (2024)
- [Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.21828/) (2025)
- [Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.14779/) (2024)
- [The challenge of hidden gifts in multi-agent reinforcement learning](https://mlsanity.com/paper/2505.20579/) (2025)
- [Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2505.08630/) (2025)

## Abstract (arXiv)

Sparse and delayed rewards pose a challenge to single agent reinforcement learning. This challenge is amplified in multi-agent reinforcement learning (MARL) where credit assignment of these rewards needs to happen not only across time, but also across agents. We propose Agent-Time Attention (ATA), a neural network model with auxiliary losses for redistributing sparse and delayed rewards in collaborative MARL. We provide a simple example that demonstrates how providing agents with their own local redistributed rewards and shared global redistributed rewards motivate different policies. We extend several MiniGrid environments, specifically MultiRoom and DoorKey, to the multi-agent sparse delayed rewards setting. We demonstrate that ATA outperforms various baselines on many instances of these environments. Source code of the experiments is available at https://github.com/jshe/agent-time-attention.