mlsanityData to 7 Oct 2026

Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization

arXiv 2504.21063 ↗First posted 29 Apr 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data while preserving privacy. Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt. However, such a one-prompt-fits-all learning paradigm typically leads to performance degradation on personalized samples. Although the mixture of experts (MoE) offers a promising solution for specialization, existing MoE-based methods suffer from coarse image-level expert assignment and high communication costs from parameterized routers. To address these limitations, we propose TRIP, a Token-level prompt mixture with parameter-free routing framework for FedDG, which treats multiple prompts as distinct experts. Unlike existing image-level routing designs, TRIP assigns different tokens within an image to specific experts. To ensure communication efficiency, TRIP incorporates a parameter-free routing mechanism based on token clustering and optimal transport. The instance-specific prompt is then synthesized by aggregating experts, weighted by the number of tokens assigned to each. Additionally, TRIP develops an unbiased learning strategy for prompt experts, leveraging the VLM's zero-shot generalization capability. Extensive experiments across four benchmarks demonstrate that TRIP achieves optimal generalization results, with communication of only 1K parameters per round. Our code is available at https://github.com/GongShuai8210/TRIP.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 53 of its 77 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

53 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  11. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  12. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
38 more
  1. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  6. 2020Federated Learning with Matched Averagingcited by 564
  7. 2021Domain Generalization with MixStylecited by 563
  8. 2015Optimal Transport for Domain Adaptationcited by 457
  9. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  10. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  11. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  12. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  13. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  14. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 322
  15. 2021A Fourier-based Framework for Domain Generalizationcited by 287
  16. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
  17. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  18. 2024Mixture of LoRA Expertscited by 179
  19. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 160
  20. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  21. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  22. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  23. 2021MixStyle Neural Networks for Domain Generalization and Adaptationcited by 64
  24. 2022Prompt Vision Transformer for Domain Generalizationcited by 52
  25. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  26. 2021Federated Learning with Domain Generalizationcited by 41
  27. 2022Federated Domain Generalization for Image Recognition via Cross-Client Style Transfercited by 40
  28. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  29. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29
  30. 2024Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversitycited by 28
  31. 2021Federated Mixture of Expertscited by 27
  32. 2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
  33. 2023MADG: Margin-based Adversarial Learning for Domain Generalizationcited by 22
  34. 2023StableFDG: Style and Attention Based Learning for Federated Domain Generalizationcited by 21
  35. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitationscited by 20
  36. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 16
  37. 2024HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalizationcited by 8
  38. 2023Towards Instance-adaptive Inference for Federated Learningcited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Federated Domain Generalization via Prompt Learning and Aggregation
  2. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning
  3. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
  4. 2025FLIP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
  5. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
  6. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  7. 2024Routers in Vision Mixture of Experts: An Empirical Study
  8. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion