mlsanityData to 14 Oct 2026

Distributed Learning over Networks with Graph-Attention-Based Personalization

arXiv 2305.13041 ↗First posted 22 May 2023cs.DC cs.LG eess.SPcited by 6

Not on the map. Why?

Abstract (arXiv)

In conventional distributed learning over a network, multiple agents collaboratively build a common machine learning model. However, due to the underlying non-i.i.d. data distribution among agents, the unified learning model becomes inefficient for each agent to process its locally accessible data. To address this problem, we propose a graph-attention-based personalized training algorithm (GATTA) for distributed deep learning. The GATTA enables each agent to train its local personalized model while exploiting its correlation with neighboring nodes and utilizing their useful information for aggregation. In particular, the personalized model in each agent is composed of a global part and a node-specific part. By treating each agent as one node in a graph and the node-specific parameters as its features, the benefits of the graph attention mechanism can be inherited. Namely, instead of aggregation based on averaging, it learns the specific weights for different neighboring nodes without requiring prior knowledge about the graph structure or the neighboring nodes' data distribution. Furthermore, relying on the weight-learning procedure, we develop a communication-efficient GATTA by skipping the transmission of information with small aggregation weights. Additionally, we theoretically analyze the convergence properties of GATTA for non-convex loss functions. Numerical results validate the excellent performances of the proposed algorithms in terms of convergence and communication cost.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

46 of 55 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Graph Attention Networkscited by 8,923
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  5. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2018Federated Learning with Non-IID Datacited by 1,108
  8. 2018LEAF: A Benchmark for Federated Settingscited by 792
  9. 2017Federated Multi-Task Learningcited by 672
  10. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  11. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  12. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
34 more
  1. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  2. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  3. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  4. 2020Adaptive Personalized Federated Learningcited by 358
  5. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  6. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  7. 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
  8. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  9. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  10. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  11. 2019Federated Evaluation of On-device Personalizationcited by 147
  12. 2020Survey of Personalization Techniques for Federated Learningcited by 145
  13. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  14. 2018D2: Decentralized Training over Decentralized Datacited by 140
  15. 2020Graph Convolutional Networks for Hyperspectral Image Classificationcited by 113
  16. 2020One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysiscited by 112
  17. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
  18. 2020Convergence Time Optimization for Federated Learning over Wireless Networkscited by 81
  19. 2020HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learningcited by 74
  20. 2021Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modelingcited by 69
  21. 2018An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixingcited by 66
  22. 2017Collaborative Deep Learning in Fixed Topology Networkscited by 60
  23. 2021Decentralized Federated Learning: Balancing Communication and Computing Costscited by 55
  24. 2020An improved convergence analysis for decentralized online stochastic non-convex optimizationcited by 53
  25. 2021Cluster-driven Graph Federated Learning over Multiple Domainscited by 44
  26. 2019Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphscited by 36
  27. 2019Multi-Agent Actor-Critic with Hierarchical Graph Attention Networkcited by 33
  28. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 33
  29. 2019Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classificationcited by 17
  30. 2020Decentralized Deep Learning using Momentum-Accelerated Consensuscited by 14
  31. 2021Resource Rationing for Wireless Federated Learning: Concept, Benefits, and Challengescited by 4
  32. 2022Personalized Federated Learning with Exact Stochastic Gradient Descentcited by 4
  33. 2020Distributed ADMM with Synergetic Communication and Computationcited by 3
  34. 2019Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 2.

6 papers.

  1. 2023A Survey on Trustworthy Edge Intelligence: From Security and Reliability To Transparency and Sustainabilitycited by 7
  2. 2025Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneitycited by 1
  3. 2025Sheaf-Based Decentralized Multimodal Learning for Next-Generation Wireless Communication Systemscited by 1
  4. 2025Federated Learning with Enhanced Privacy via Model Splitting and Random Client Participationcited by 0
  5. 2026Communication-Efficient Personalized Adaptation via Federated-Local Model Mergingcited by 0
  6. 2026Semantic-based Distributed Learning for Diverse and Discriminative Representationscited by 0