# FedFormer: Contextual Federation with Attention in Reinforcement Learning

arXiv: https://arxiv.org/abs/2205.13697 · HTML: https://mlsanity.com/paper/2205.13697/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2108.11887/) (2021)
- [Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee](https://mlsanity.com/paper/2110.14074/) (2021)
- [Federated Deep Reinforcement Learning](https://mlsanity.com/paper/1901.08277/) (2019)
- [Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG](https://mlsanity.com/paper/1811.07029/) (2018)
- [Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services](https://mlsanity.com/paper/1810.04118/) (2018)
- [Federated Deep Reinforcement Learning for the Distributed Control of NextG Wireless Networks](https://mlsanity.com/paper/2112.03465/) (2021)

## Built on by (most cited first, 8 of 8)

- [Federated Ensemble-Directed Offline Reinforcement Learning](https://mlsanity.com/paper/2305.03097/) (2023)
- [FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data](https://mlsanity.com/paper/2512.02350/) (2025)
- [Task-agnostic Decision Transformer for Multi-type Agent Control with Federated Split Training](https://mlsanity.com/paper/2405.13445/) (2024)
- [Federated Hierarchical Reinforcement Learning for Adaptive Traffic Signal Control](https://mlsanity.com/paper/2504.05553/) (2025)
- [PPS-QMIX: Periodically Parameter Sharing for Accelerating Convergence of Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.02635/) (2024)
- [Trading-off Accuracy and Communication Cost in Federated Learning](https://mlsanity.com/paper/2503.14246/) (2025)
- [Privacy Preserved Federated Learning with Attention-Based Aggregation for Biometric Recognition](https://mlsanity.com/paper/2510.01113/) (2025)
- [Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge](https://mlsanity.com/paper/2511.11585/) (2025)

## Abstract (arXiv)

A core issue in multi-agent federated reinforcement learning is defining how to aggregate insights from multiple agents. This is commonly done by taking the average of each participating agent's model weights into one common model (FedAvg). We instead propose FedFormer, a novel federation strategy that utilizes Transformer Attention to contextually aggregate embeddings from models originating from different learner agents. In so doing, we attentively weigh the contributions of other agents with respect to the current agent's environment and learned relationships, thus providing a more effective and efficient federation. We evaluate our methods on the Meta-World environment and find that our approach yields significant improvements over FedAvg and non-federated Soft Actor-Critic single-agent methods. Our results compared to Soft Actor-Critic show that FedFormer achieves higher episodic return while still abiding by the privacy constraints of federated learning. Finally, we also demonstrate improvements in effectiveness with increased agent pools across all methods in certain tasks. This is contrasted by FedAvg, which fails to make noticeable improvements when scaled.