mlsanityData to 7 Oct 2026

CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data

arXiv 2312.08648 ↗First posted 14 Dec 2023cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) provides a decentralized machine learning paradigm where a server collaborates with a group of clients to learn a global model without accessing the clients' data. User heterogeneity is a significant challenge for FL, which together with the class-distribution imbalance further enhances the difficulty of FL. Great progress has been made in large vision-language models, such as Contrastive Language-Image Pre-training (CLIP), which paves a new way for image classification and object recognition. Inspired by the success of CLIP on few-shot and zero-shot learning, we use CLIP to optimize the federated learning between server and client models under its vision-language supervision. It is promising to mitigate the user heterogeneity and class-distribution balance due to the powerful cross-modality representation and rich open-vocabulary prior knowledge. In this paper, we propose the CLIP-guided FL (CLIP2FL) method on heterogeneous and long-tailed data. In CLIP2FL, the knowledge of the off-the-shelf CLIP model is transferred to the client-server models, and a bridge is built between the client and server. Specifically, for client-side learning, knowledge distillation is conducted between client models and CLIP to improve the ability of client-side feature representation. For server-side learning, in order to mitigate the heterogeneity and class-distribution imbalance, we generate federated features to retrain the server model. A prototype contrastive learning with the supervision of the text encoder of CLIP is introduced to generate federated features depending on the client-side gradients, and they are used to retrain a balanced server classifier.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

30 of 50 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  10. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
18 more
  1. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  4. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  5. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  6. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  7. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  8. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  9. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  10. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  11. 2022Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Featurescited by 63
  12. 2020Federated learning with class imbalance reductioncited by 52
  13. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  14. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  15. 2020Fed-Focal Loss for imbalanced data classification in Federated Learningcited by 26
  16. 2021Behavior Mimics Distribution: Combining Individual and Group Behaviors for Federated Learningcited by 7
  17. 2023Towards Instance-adaptive Inference for Federated Learningcited by 7
  18. 2023Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Searchcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 4.

9 papers.

  1. 2024Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversitycited by 28
  2. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  3. 2024Accurately Predicting Probabilities of Safety-Critical Rare Events for Intelligent Systemscited by 3
  4. 2024MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learningcited by 3
  5. 2024Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Modelcited by 2
  6. 2025CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Modelcited by 1
  7. 2025Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Modelscited by 1
  8. 2024FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learningcited by 0
  9. 2025FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarioscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Federated CLIP for Resource-Efficient Heterogeneous Medical Image Classification
  2. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  3. 2024Lightweight Unsupervised Federated Learning with Pretrained Vision Language Model
  4. 2025FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models
  5. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classification
  6. 2025FAA-CLIP: Federated Adversarial Adaptation of CLIP
  7. 2025Zero-Shot Decentralized Federated Learning
  8. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning