mlsanityData to 7 Oct 2026

Federated Vision-Language-Recommendation with Personalized Fusion

arXiv 2410.08478 ↗First posted 11 Oct 2024cs.IR cs.AI cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Applying large pre-trained Vision-Language Models to recommendation is a burgeoning field, a direction we term Vision-Language-Recommendation (VLR). Bringing VLR to user-oriented on-device intelligence within a federated learning framework is a crucial step for enhancing user privacy and delivering personalized experiences. This paper introduces FedVLR, a federated VLR framework specially designed for user-specific personalized fusion of vision-language representations. At its core is a novel bi-level fusion mechanism: The server-side multi-view fusion module first generates a diverse set of pre-fused multimodal views. Subsequently, each client employs a user-specific mixture-of-expert mechanism to adaptively integrate these views based on individual user interaction history. This designed lightweight personalized fusion module provides an efficient solution to implement a federated VLR system. The effectiveness of our proposed FedVLR has been validated on seven benchmark datasets.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 50 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

28 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method29 Sep

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2015VBPR: Visual Bayesian Personalized Ranking from Implicit Feedbackcited by 151
  8. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  9. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  10. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  11. 2022Bootstrap Latent Representations for Multi-modal Recommendationcited by 55
  12. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 45
15 more
  1. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  2. 2023Multi-View Graph Convolutional Network for Multimedia Recommendationcited by 32
  3. 2023MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendationcited by 30
  4. 2022LightFR: Lightweight Federated Recommendation with Privacy-preserving Matrix Factorizationcited by 26
  5. 2024Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyondcited by 25
  6. 2022Personalized Federated Recommendation via Joint Representation Learning, User Clustering, and Model Adaptationcited by 24
  7. 2024Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Surveycited by 24
  8. 2023HeteFedRec: Federated Recommender Systems with Model Heterogeneitycited by 20
  9. 2024On-Device Recommender Systems: A Comprehensive Surveycited by 20
  10. 2023NineRec: A Benchmark Dataset Suite for Evaluating Transferable Recommendationcited by 18
  11. 2024Federated Adaptation for Foundation Model-based Recommendationscited by 14
  12. 2019User Diverse Preference Modeling by Multimodal Attentive Metric Learningcited by 10
  13. 2024Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approachcited by 9
  14. 2023Universal Multi-modal Multi-domain Pre-trained Recommendationcited by 2
  15. 2024Towards Bridging the Cross-modal Semantic Gap for Multi-modal Recommendationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  2. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  3. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
  4. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models
  5. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
  6. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  7. 2024Harmonizing Generalization and Personalization in Federated Prompt Learning
  8. 2026UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity