# Homogeneous Learning: Self-Attention Decentralized Deep Learning

arXiv: https://arxiv.org/abs/2110.05290 · HTML: https://mlsanity.com/paper/2110.05290/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 8 of 8)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [A Blockchain-based Decentralized Federated Learning Framework with Committee Consensus](https://mlsanity.com/paper/2004.00773/) (2020)
- [Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems](https://mlsanity.com/paper/1901.06455/) (2019)
- [Detailed comparison of communication efficiency of split learning and federated learning](https://mlsanity.com/paper/1909.09145/) (2019)
- [Fusion of Federated Learning and Industrial Internet of Things: A Survey](https://mlsanity.com/paper/2101.00798/) (2021)

## Built on by (most cited first, 4 of 4)

- [Feature Distribution Matching for Federated Domain Generalization](https://mlsanity.com/paper/2203.11635/) (2022)
- [FedMCSA: Personalized Federated Learning via Model Components Self-Attention](https://mlsanity.com/paper/2208.10731/) (2022)
- [Meta Learning in Decentralized Neural Networks: Towards More General AI](https://mlsanity.com/paper/2302.01020/) (2023)
- [Meta Neural Coordination](https://mlsanity.com/paper/2305.12109/) (2023)

## Abstract (arXiv)

Federated learning (FL) has been facilitating privacy-preserving deep learning in many walks of life such as medical image classification, network intrusion detection, and so forth. Whereas it necessitates a central parameter server for model aggregation, which brings about delayed model communication and vulnerability to adversarial attacks. A fully decentralized architecture like Swarm Learning allows peer-to-peer communication among distributed nodes, without the central server. One of the most challenging issues in decentralized deep learning is that data owned by each node are usually non-independent and identically distributed (non-IID), causing time-consuming convergence of model training. To this end, we propose a decentralized learning model called Homogeneous Learning (HL) for tackling non-IID data with a self-attention mechanism. In HL, training performs on each round's selected node, and the trained model of a node is sent to the next selected node at the end of each round. Notably, for the selection, the self-attention mechanism leverages reinforcement learning to observe a node's inner state and its surrounding environment's state, and find out which node should be selected to optimize the training. We evaluate our method with various scenarios for an image classification task. The result suggests that HL can produce a better performance compared with standalone learning and greatly reduce both the total training rounds by 50.8% and the communication cost by 74.6% compared with random policy-based decentralized learning for training on non-IID data.