# Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG

arXiv: https://arxiv.org/abs/1811.07029 · HTML: https://mlsanity.com/paper/1811.07029/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 20 of 20)

- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2108.11887/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applications](https://mlsanity.com/paper/2206.07922/) (2022)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation](https://mlsanity.com/paper/2403.08282/) (2024)
- [SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition](https://mlsanity.com/paper/2111.08857/) (2021)
- [Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model](https://mlsanity.com/paper/2404.04619/) (2024)
- [Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?](https://mlsanity.com/paper/2305.17352/) (2023)
- [Distributed Reinforcement Learning for Robot Teams: A Review](https://mlsanity.com/paper/2204.03516/) (2022)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [Learning Altruistic Behaviours in Reinforcement Learning without External Rewards](https://mlsanity.com/paper/2107.09598/) (2021)
- [FedFormer: Contextual Federation with Attention in Reinforcement Learning](https://mlsanity.com/paper/2205.13697/) (2022)
- [Attention Actor-Critic algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning](https://mlsanity.com/paper/2101.02349/) (2021)
- [Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.06042/) (2021)
- [Efficient Policy Generation in Multi-Agent Systems via Hypergraph Neural Network](https://mlsanity.com/paper/2203.03265/) (2022)
- [Triple-BERT: Do We Really Need MARL for Order Dispatch on Ride-Sharing Platforms?](https://mlsanity.com/paper/2510.03257/) (2025)
- [SA-MATD3:Self-attention-based multi-agent continuous control method in cooperative environments](https://mlsanity.com/paper/2107.00284/) (2021)
- [Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus](https://mlsanity.com/paper/2604.13472/) (2026)
- [An Algorithm For Adversary Aware Decentralized Networked MARL](https://mlsanity.com/paper/2305.05573/) (2023)
- [Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective](https://mlsanity.com/paper/2510.09937/) (2025)

## Abstract (arXiv)

Modelling and exploiting teammates' policies in cooperative multi-agent systems have long been an interest and also a big challenge for the reinforcement learning (RL) community. The interest lies in the fact that if the agent knows the teammates' policies, it can adjust its own policy accordingly to arrive at proper cooperations; while the challenge is that the agents' policies are changing continuously due to they are learning concurrently, which imposes difficulty to model the dynamic policies of teammates accurately. In this paper, we present ATTention Multi-Agent Deep Deterministic Policy Gradient (ATT-MADDPG) to address this challenge. ATT-MADDPG extends DDPG, a single-agent actor-critic RL method, with two special designs. First, in order to model the teammates' policies, the agent should get access to the observations and actions of teammates. ATT-MADDPG adopts a centralized critic to collect such information. Second, to model the teammates' policies using the collected information in an effective way, ATT-MADDPG enhances the centralized critic with an attention mechanism. This attention mechanism introduces a special structure to explicitly model the dynamic joint policy of teammates, making sure that the collected information can be processed efficiently. We evaluate ATT-MADDPG on both benchmark tasks and the real-world packet routing tasks. Experimental results show that it not only outperforms the state-of-the-art RL-based methods and rule-based methods by a large margin, but also achieves better performance in terms of scalability and robustness.