# FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks

arXiv: https://arxiv.org/abs/2111.11066 · HTML: https://mlsanity.com/paper/2111.11066/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 89 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 30 of 73)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Analyzing Federated Learning through an Adversarial Lens](https://mlsanity.com/paper/1811.12470/) (2018)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)

## Built on by (most cited first, 30 of 32)

- [FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings](https://mlsanity.com/paper/2210.04620/) (2022)
- [Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2206.02465/) (2022)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data](https://mlsanity.com/paper/2211.09299/) (2022)
- [On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond](https://mlsanity.com/paper/2206.05187/) (2022)
- [On the Convergence of Clustered Federated Learning](https://mlsanity.com/paper/2202.06187/) (2022)
- [Personalized Federated Learning With Graph](https://mlsanity.com/paper/2203.00829/) (2022)
- [Towards Federated Learning against Noisy Labels via Local Self-Regularization](https://mlsanity.com/paper/2208.12807/) (2022)
- [FLAIR: Federated Learning Annotated Image Repository](https://mlsanity.com/paper/2207.08869/) (2022)
- [Federated Learning for Computer Vision](https://mlsanity.com/paper/2308.13558/) (2023)
- [FedImpro: Measuring and Improving Client Update in Federated Learning](https://mlsanity.com/paper/2402.07011/) (2024)
- [Differentially Private CutMix for Split Learning with Vision Transformer](https://mlsanity.com/paper/2210.15986/) (2022)
- [FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion](https://mlsanity.com/paper/2410.20380/) (2024)
- [A Privacy-Preserving Hybrid Federated Learning Framework for Financial Crime Detection](https://mlsanity.com/paper/2302.03654/) (2023)
- [FLAME: Federated Learning Across Multi-device Environments](https://mlsanity.com/paper/2202.08922/) (2022)
- [FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things](https://mlsanity.com/paper/2310.00109/) (2023)
- [Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition](https://mlsanity.com/paper/2310.15165/) (2023)
- [Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings](https://mlsanity.com/paper/2306.08586/) (2023)
- [MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes](https://mlsanity.com/paper/2404.09232/) (2024)
- [COALA: A Practical and Vision-Centric Federated Learning Platform](https://mlsanity.com/paper/2407.16560/) (2024)
- [Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beat](https://mlsanity.com/paper/2309.03237/) (2023)
- [Federated Object Detection for Quality Inspection in Shared Production](https://mlsanity.com/paper/2306.17645/) (2023)
- [Redefining non-IID Data in Federated Learning for Computer Vision Tasks: Migrating from Labels to Embeddings for Task-Specific Data Distributions](https://mlsanity.com/paper/2503.14553/) (2025)
- [Equitable Federated Learning with NCA](https://mlsanity.com/paper/2506.21735/) (2025)
- [Split Learning in Computer Vision for Semantic Segmentation Delay Minimization](https://mlsanity.com/paper/2412.14272/) (2024)
- [Personalized Federated Learning via Learning Dynamic Graphs](https://mlsanity.com/paper/2503.05474/) (2025)
- [ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations](https://mlsanity.com/paper/2305.19146/) (2023)
- [Unlocking the Potential of Federated Learning for Deeper Models](https://mlsanity.com/paper/2306.02701/) (2023)
- [Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation](https://mlsanity.com/paper/2509.20680/) (2025)
- [Learning Color Grading, No Photo Sharing: Federated Aesthetic Preference Learning for Personalized Image Enhancement](https://mlsanity.com/paper/2607.27659/) (2026)

## Abstract (arXiv)

Federated Learning (FL) is a distributed learning paradigm that can learn a global or personalized model from decentralized datasets on edge devices. However, in the computer vision domain, model performance in FL is far behind centralized training due to the lack of exploration in diverse tasks with a unified FL framework. FL has rarely been demonstrated effectively in advanced computer vision tasks such as object detection and image segmentation. To bridge the gap and facilitate the development of FL for computer vision tasks, in this work, we propose a federated learning library and benchmarking framework, named FedCV, to evaluate FL on the three most representative computer vision tasks: image classification, image segmentation, and object detection. We provide non-I.I.D. benchmarking datasets, models, and various reference FL algorithms. Our benchmark study suggests that there are multiple challenges that deserve future exploration: centralized training tricks may not be directly applied to FL; the non-I.I.D. dataset actually downgrades the model accuracy to some degree in different tasks; improving the system efficiency of federated training is challenging given the huge number of parameters and the per-client memory cost. We believe that such a library and benchmark, along with comparable evaluation settings, is necessary to make meaningful progress in FL on computer vision tasks. FedCV is publicly available: https://github.com/FedML-AI/FedCV.