mlsanityData to 14 Oct 2026

BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

arXiv 2610.12907 ↗First posted 9 Oct 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

30 of 43 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions21 Aug
  2. 2025SemPT: Semantic Prompt Tuning for Vision-Language Models14 Aug
  3. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models7 Jul
  4. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts21 May
  5. 2025Bayesian Principles Improve Prompt Learning In Vision-Language Models19 Apr
  6. 2025DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models17 Mar
  7. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models14 Oct
  8. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method29 Sep
  9. 2024Efficient Test-Time Prompt Tuning for Vision-Language Models11 Aug
  10. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  11. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models11 Apr
  12. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model30 Nov
4 more
  1. 2022Learning Domain Invariant Prompt for Vision-Language Models8 Dec
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  5. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  6. 2020Federated Visual Classification with Real-World Data Distributioncited by 128
  7. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  8. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 46
  9. 2025MMRL: Multi-Modal Representation Learning for Vision-Language Modelscited by 39
  10. 2023Progressive Visual Prompt Learning with Contrastive Feature Re-formationcited by 36
  11. 2025Multi-Modal Multi-Task (M3T) Federated Foundation Models for Embodied AI: Potentials and Challenges for Edge Integrationcited by 6
  12. 2025Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MRcited by 3
2 more
  1. 2024TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiencycited by 2
  2. 2025Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architecturescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  2. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  3. 2024Federated Vision-Language-Recommendation with Personalized Fusion
  4. 2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning
  5. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
  6. 2026UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
  7. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models
  8. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics