# Federated Vision-Language-Recommendation with Personalized Fusion

arXiv: https://arxiv.org/abs/2410.08478 · HTML: https://mlsanity.com/paper/2410.08478/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model](https://mlsanity.com/paper/2208.11625/) (2022)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [Diagnosing and Rectifying Vision Models using Language](https://mlsanity.com/paper/2302.04269/) (2023)
- [Bootstrap Latent Representations for Multi-modal Recommendation](https://mlsanity.com/paper/2207.05969/) (2022)
- [VIP5: Towards Multimodal Foundation Models for Recommendation](https://mlsanity.com/paper/2305.14302/) (2023)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [Multi-View Graph Convolutional Network for Multimedia Recommendation](https://mlsanity.com/paper/2308.03588/) (2023)
- [MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation](https://mlsanity.com/paper/2308.11175/) (2023)
- [LightFR: Lightweight Federated Recommendation with Privacy-preserving Matrix Factorization](https://mlsanity.com/paper/2206.11743/) (2022)
- [Personalized Federated Recommendation via Joint Representation Learning, User Clustering, and Model Adaptation](https://mlsanity.com/paper/2208.09375/) (2022)
- [Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond](https://mlsanity.com/paper/2403.10667/) (2024)
- [Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey](https://mlsanity.com/paper/2404.00621/) (2024)
- [Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method](https://mlsanity.com/paper/2409.19610/) (2024)
- [NineRec: A Benchmark Dataset Suite for Evaluating Transferable Recommendation](https://mlsanity.com/paper/2309.07705/) (2023)
- [HeteFedRec: Federated Recommender Systems with Model Heterogeneity](https://mlsanity.com/paper/2307.12810/) (2023)
- [On-Device Recommender Systems: A Comprehensive Survey](https://mlsanity.com/paper/2401.11441/) (2024)
- [Federated Adaptation for Foundation Model-based Recommendations](https://mlsanity.com/paper/2405.04840/) (2024)
- [User Diverse Preference Modeling by Multimodal Attentive Metric Learning](https://mlsanity.com/paper/1908.07738/) (2019)
- [Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach](https://mlsanity.com/paper/2408.08931/) (2024)
- [Universal Multi-modal Multi-domain Pre-trained Recommendation](https://mlsanity.com/paper/2311.01831/) (2023)
- [Towards Bridging the Cross-modal Semantic Gap for Multi-modal Recommendation](https://mlsanity.com/paper/2407.05420/) (2024)

## Built on by (most cited first, 1 of 1)

- [A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendation](https://mlsanity.com/paper/2506.11563/) (2025)

## Abstract (arXiv)

Applying large pre-trained Vision-Language Models to recommendation is a burgeoning field, a direction we term Vision-Language-Recommendation (VLR). Bringing VLR to user-oriented on-device intelligence within a federated learning framework is a crucial step for enhancing user privacy and delivering personalized experiences. This paper introduces FedVLR, a federated VLR framework specially designed for user-specific personalized fusion of vision-language representations. At its core is a novel bi-level fusion mechanism: The server-side multi-view fusion module first generates a diverse set of pre-fused multimodal views. Subsequently, each client employs a user-specific mixture-of-expert mechanism to adaptively integrate these views based on individual user interaction history. This designed lightweight personalized fusion module provides an efficient solution to implement a federated VLR system. The effectiveness of our proposed FedVLR has been validated on seven benchmark datasets.