mlsanityData to 7 Oct 2026

Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models

arXiv 2610.01625 ↗First posted 1 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Federated parameter-efficient fine-tuning enables distributed clients to adapt pretrained vision-language models without sharing raw data or updating the full backbone. Its effectiveness, however, is limited by domain heterogeneity across clients. Existing personalized methods separate globally shared knowledge from client-specific style, but they largely treat each domain as a class-agnostic transformation. We show that this abstraction is insufficient: the cross-domain displacement associated with a fixed domain varies across semantic classes, and only a subset of these class-domain residuals damages the image-text decision margin. We therefore propose Margin-Oriented Semantic-Appearance Interaction Correction (MOSAIC), which first constructs a decision-aware harmfulness score that measures whether a training-derived class-domain residual favors a competing text prototype over the true class. It then models fine-grained class-domain interactions with a low-rank residual adapter whose class factors and residual basis are globally shared while domain factors remain client-private. An image-conditioned gate further controls candidate-wise correction, and harmful-pair-aware reweighting prioritizes decision-relevant residuals during local optimization. Extensive experiments on Office31, OfficeHome, and DomainNet100 demonstrate that MOSAIC consistently improves macro-client top-1 accuracy across all evaluated domain-shift and joint domain-label-shift settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 29 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

14 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning16 May
  2. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  6. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  8. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  9. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  10. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  11. 2025Federated Domain Generalization with Data-free On-server Matching Gradientcited by 7
  12. 2025Federated Domain Generalization with Domain-specific Soft Prompts Generationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
  2. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  3. 2026Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
  4. 2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models
  5. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  6. 2025Beyond Retraining: Training-Free Unknown Class Filtering for Source-Free Open Set Domain Adaptation of Vision-Language Models
  7. 2025On the Domain Robustness of Contrastive Vision-Language Models
  8. 2026Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models