# Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability

arXiv: https://arxiv.org/abs/2301.01649 · HTML: https://mlsanity.com/paper/2301.01649/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2023.

## Built on by (most cited first, 8 of 8)

- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?](https://mlsanity.com/paper/2305.17352/) (2023)
- [AgentMixer: Multi-Agent Correlated Policy Factorization](https://mlsanity.com/paper/2401.08728/) (2024)
- [Towards Fault Tolerance in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.00534/) (2024)
- [Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments](https://mlsanity.com/paper/2407.04974/) (2024)
- [Swarm Behavior Cloning](https://mlsanity.com/paper/2412.07617/) (2024)
- [DC-Ada: Reward-Only Decentralized Sensor Adaptation for Heterogeneous Multi-Robot Teams](https://mlsanity.com/paper/2604.03905/) (2026)
- [HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2606.30966/) (2026)

## Abstract (arXiv)

Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attention-based Embeddings of Recurrence In multi-Agent Learning (AERIAL) to approximate value functions under stochastic partial observability. AERIAL replaces the true state with a learned representation of multi-agent recurrence, considering more accurate information about decentralized agent decisions than state-based CTDE. We then introduce MessySMAC, a modified version of SMAC with stochastic observations and higher variance in initial states, to provide a more general and configurable benchmark regarding stochastic partial observability. We evaluate AERIAL in Dec-Tiger as well as in a variety of SMAC and MessySMAC maps, and compare the results with state-based CTDE. Furthermore, we evaluate the robustness of AERIAL and state-based CTDE against various stochasticity configurations in MessySMAC.