mlsanityData to 14 Oct 2026

How (Mis)calibrated is your Federated CLIP and what to do about it?

arXiv 2512.04305 ↗First posted 3 Dec 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) such as CLIP are increasingly adapted across decentralized data silos, yet the reliability of their predictions under federated learning (FL) remains largely unexplored. In this work, we present a systematic study of calibration in federated CLIP under non-IID client distributions. Our experiments reveal that widely used prompt-tuning methods consistently degrade calibration, often yielding substantially higher calibration error despite competitive recognition performance, while explicit training-time calibration regularizers provide only limited improvements. Motivated by these findings, we identify the choice of fine-tuning parameterization as a critical factor governing calibration and conduct a controlled comparison between prompt tuning and five backbone fine-tuning (BFT) strategies: AdaptFormer, LayerNorm, LoRA, VeRA, and DoRA. We find that BFT methods generally offer a more favorable accuracy-calibration trade-off than prompt tuning, although their benefits are not universal. Through extensive analysis, we show that calibration behavior is closely linked to the geometry of federated updates, residual parameterization, and the resulting client and logit drift. Across in-distribution, domain-generalization, and base-to-new evaluation settings, our results establish fine-tuning parameterization as a central design choice for building accurate and reliable federated CLIP models. Codes are available at https://github.com/mainaksingha01/FL2oRA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 79 references that are in the map. First posted on arXiv 3 Dec 2025.

Builds on

61 of 79 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models29 Apr
  2. 2025Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models11 Feb
  3. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models3 Oct
  4. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  5. 2024Open-Vocabulary Calibration for Fine-tuned CLIP7 Feb
  6. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
38 more
  1. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  2. 2019When Does Label Smoothing Help?cited by 988
  3. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  4. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  6. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  9. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  10. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  11. 2019Measuring Calibration in Deep Learningcited by 446
  12. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  13. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  14. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  15. 2024LoRA Learns Less and Forgets Lesscited by 328
  16. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  17. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  18. 2021Personalized Federated Learning using Hypernetworkscited by 212
  19. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  20. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  21. 2017Domain Adaptation for Visual Applications: A Comprehensive Surveycited by 128
  22. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  23. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  24. 2023LoRA ensembles for large language model fine-tuningcited by 67
  25. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  26. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  27. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  28. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 46
  29. 2020Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classificationcited by 37
  30. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  31. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  32. 2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networkscited by 20
  33. 2025Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)cited by 20
  34. 2022Knowledge-Aware Federated Active Learning with Non-IID Datacited by 16
  35. 2023Guiding The Last Layer in Federated Learning with Pre-Trained Modelscited by 13
  36. 2024Towards Multi-modal Transformers in Federated Learningcited by 12
  37. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  38. 2024Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensemblescited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment
  2. 2023CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data
  3. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  4. 2025CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Model
  5. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  6. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
  7. 2025FAA-CLIP: Federated Adversarial Adaptation of CLIP
  8. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models