mlsanityData to 7 Oct 2026

Structured Cooperative Learning with Graphical Model Priors

arXiv 2306.09595 ↗First posted 16 Jun 2023cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We study how to train personalized models for different tasks on decentralized devices with limited local data. We propose "Structured Cooperative Learning (SCooL)", in which a cooperation graph across devices is generated by a graphical model prior to automatically coordinate mutual learning between devices. By choosing graphical models enforcing different structures, we can derive a rich class of existing and novel decentralized learning algorithms via variational inference. In particular, we show three instantiations of SCooL that adopt Dirac distribution, stochastic block model (SBM), and attention as the prior generating cooperation graphs. These EM-type algorithms alternate between updating the cooperation graph and cooperative learning of local models. They can automatically capture the cross-task correlations among devices by only monitoring their model updating in order to optimize the cooperation graph. We evaluate SCooL and compare it with existing decentralized learning methods on an extensive set of benchmarks, on which SCooL always achieves the highest accuracy of personalized models and significantly outperforms other baselines on communication efficiency. Our code is available at https://github.com/ShuangtongLi/SCooL.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

41 of 56 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2018Group Normalizationcited by 1,514
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  9. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  10. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  11. 2020Federated Learning with Matched Averagingcited by 564
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 555
29 more
  1. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  2. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  3. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  4. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  6. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  7. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  8. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  9. 2021Personalized Federated Learning using Hypernetworkscited by 210
  10. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  11. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  12. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  13. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  14. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
  15. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  16. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  17. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  18. 2017Collaborative Deep Learning in Fixed Topology Networkscited by 60
  19. 2019Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphscited by 34
  20. 2016Gossip training for deep learningcited by 33
  21. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 32
  22. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  23. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  24. 2022Beyond spectral gap: The role of the topology in decentralized learningcited by 27
  25. 2022Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Ratecited by 25
  26. 2022Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimizationcited by 22
  27. 2023Dual Personalization on Federated Recommendationcited by 19
  28. 2022Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topologycited by 10
  29. 2016Decentralized Clustering and Linking by Networked Agentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Decentralized Multitask Learning over Learned Task Graphscited by 0