mlsanityData to 14 Oct 2026

Redefining non-IID Data in Federated Learning for Computer Vision Tasks: Migrating from Labels to Embeddings for Task-Specific Data Distributions

arXiv 2503.14553 ↗First posted 17 Mar 2025cs.CV cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has emerged as one of the prominent paradigms for distributed machine learning (ML). However, it is well-established that its performance can degrade significantly under non-IID (non-independent and identically distributed) data distributions across clients. To study this effect, the existing works predominantly emulate data heterogeneity by imposing label distribution skew across clients. In this paper, we show that label distribution skew fails to fully capture the data heterogeneity in computer vision tasks beyond classification, exposing an overlooked gap in the literature. Motivated by this, by utilizing pre-trained deep neural networks to extract task-specific data embeddings, we define task-specific data heterogeneity through the lens of each vision task and introduce a new level of data heterogeneity called embedding-based data heterogeneity. Our methodology involves clustering data points based on embeddings and distributing them among clients using the Dirichlet distribution. Through extensive experiments, we evaluate the performance of different FL methods under our revamped notion of data heterogeneity, introducing new benchmark performance measures to the literature. For instance, across seven representative computer vision tasks, our embedding-based heterogeneity formulation leads to up to around 60% increase in the observed loss under FedAvg, indicating that it more accurately exposes the performance degradation caused by data heterogeneity. We further unveil a series of open research directions that can be pursued. (Code: https://github.com/KasraBorazjani/task-perspective-het.git)

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

27 of 49 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2017Federated Multi-Task Learningcited by 672
  11. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  12. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
15 more
  1. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  2. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  3. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
  4. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  5. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  6. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  7. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  8. 2019A Principled Approach for Learning Task Similarity in Multitask Learningcited by 32
  9. 2021FedCV: A Federated Learning Framework for Diverse Computer Vision Taskscited by 32
  10. 2021SplitAVG: A heterogeneity-aware federated deep learning method for medical imagingcited by 19
  11. 2023FedHCA2: Towards Hetero-Client Federated Multi-Task Learningcited by 17
  12. 2023MAS: Towards Resource-Efficient Federated Multiple-Task Learningcited by 9
  13. 2023Handling Data Heterogeneity via Architectural Design for Federated Visual Recognitioncited by 8
  14. 2021Fisher Task Distance and Its Application in Neural Architecture Searchcited by 7
  15. 2024Multi-Modal Federated Learning for Cancer Staging over Non-IID Datasets with Unbalanced Modalitiescited by 6

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MRcited by 3
  2. 2025Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detectioncited by 0
  3. 2025Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challengescited by 0