mlsanityData to 7 Oct 2026

Tackling Feature and Sample Heterogeneity in Decentralized Multi-Task Learning: A Sheaf-Theoretic Approach

arXiv 2502.01145 ↗First posted 3 Feb 2025cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Federated multi-task learning (FMTL) aims to simultaneously learn multiple related tasks across clients without sharing sensitive raw data. However, in the decentralized setting, existing FMTL frameworks are limited in their ability to capture complex task relationships and handle feature and sample heterogeneity across clients. To address these challenges, we introduce a novel sheaf-theoretic-based approach for FMTL. By representing client relationships using cellular sheaves, our framework can flexibly model interactions between heterogeneous client models. We formulate the sheaf-based FMTL optimization problem using sheaf Laplacian regularization and propose the Sheaf-FMTL algorithm to solve it. We show that the proposed framework provides a unified view encompassing many existing federated learning (FL) and FMTL approaches. Furthermore, we prove that our proposed algorithm, Sheaf-FMTL, achieves a sublinear convergence rate in line with state-of-the-art decentralized FMTL algorithms. Extensive experiments show that although Sheaf-FMTL introduces computational and storage overhead due to the management of interaction maps, it achieves substantial communication savings in terms of transmitted bits when compared to decentralized FMTL baselines. This trade-off makes Sheaf-FMTL especially suitable for cross-silo FL scenarios, where managing model heterogeneity and ensuring communication efficiency are essential, and where clients have adequate computational resources.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

40 of 57 references.

References · 40

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  10. 2017Federated Multi-Task Learningcited by 669
  11. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  12. 2020Personalized Federated Learning with Moreau Envelopescited by 612
28 more
  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  2. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  3. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  4. 2019Federated Learning with Personalization Layerscited by 533
  5. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  6. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  7. 2020Adaptive Personalized Federated Learningcited by 353
  8. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  9. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  11. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  12. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  13. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  14. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  15. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  16. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  17. 2019Federated Evaluation of On-device Personalizationcited by 147
  18. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  19. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
  20. 2021Distributionally Robust Federated Averagingcited by 104
  21. 2019Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applicationscited by 64
  22. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
  23. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  24. 2021A New Look and Convergence Rate of Federated Multi-Task Learning with Laplacian Regularizationcited by 27
  25. 2022FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcited by 23
  26. 2021Clustered Federated Learning via Generalized Total Variation Minimizationcited by 7
  27. 2023Distributed Personalized Empirical Risk Minimizationcited by 6
  28. 2022Diffusion of Information on Networked Lattices by Gossipcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Semantic Communication meets System 2 ML: How Abstraction, Compositionality and Emergent Languages Shape Intelligencecited by 2
  2. 2025SheafAlign: A Sheaf-theoretic Framework for Decentralized Multimodal Alignmentcited by 2
  3. 2026Toward Enhancing Representation Learning in Federated Multi-Task Settingscited by 2
  4. 2026Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensuscited by 2
  5. 2025Resilient-native and Intelligent NextG Systemscited by 1
  6. 2025Sheaf-Based Decentralized Multimodal Learning for Next-Generation Wireless Communication Systemscited by 1
  7. 2025Learning to Collaborate Over Graphs: A Selective Federated Multi-Task Learning Approachcited by 0
  8. 2026Prototype-Regularized Federated Learning for Cross-Domain Aspect Sentiment Triplet Extractioncited by 0
  9. 2026MuCALD-SplitFed: Causal-Latent Diffusion for Privacy-Preserving Multi-Task Split-Federated Medical Image Segmentationcited by 0
  10. 2026Sheaf-Based Federated Representation Learningcited by 0