# Bridging Domain Gaps between Pretrained Multimodal Models and Recommendations

arXiv: https://arxiv.org/abs/2502.15542 · HTML: https://mlsanity.com/paper/2502.15542/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient](https://mlsanity.com/paper/2405.15304/) (2024)
- [Number it: Temporal Grounding Videos like Flipping Manga](https://mlsanity.com/paper/2411.10332/) (2024)
- [Exploring Diverse In-Context Configurations for Image Captioning](https://mlsanity.com/paper/2305.14800/) (2023)
- [A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation](https://mlsanity.com/paper/2211.06924/) (2022)
- [Multi-View Graph Convolutional Network for Multimedia Recommendation](https://mlsanity.com/paper/2308.03588/) (2023)
- [MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation](https://mlsanity.com/paper/2308.11175/) (2023)
- [Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights](https://mlsanity.com/paper/2305.15036/) (2023)
- [TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback](https://mlsanity.com/paper/2206.06190/) (2022)
- [Bootstrapping User and Item Representations for One-Class Collaborative Filtering](https://mlsanity.com/paper/2105.06323/) (2021)
- [PromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning](https://mlsanity.com/paper/2402.17188/) (2024)
- [Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation](https://mlsanity.com/paper/2412.14978/) (2024)

## Abstract (arXiv)

With the explosive growth of multimodal content online, pre-trained visual-language models have shown great potential for multimodal recommendation. However, while these models achieve decent performance when applied in a frozen manner, surprisingly, due to significant domain gaps (e.g., feature distribution discrepancy and task objective misalignment) between pre-training and personalized recommendation, adopting a joint training approach instead leads to performance worse than baseline. Existing approaches either rely on simple feature extraction or require computationally expensive full model fine-tuning, struggling to balance effectiveness and efficiency. To tackle these challenges, we propose Parameter-efficient Tuning for Multimodal Recommendation (PTMRec), a novel framework that bridges the domain gap between pre-trained models and recommendation systems through a knowledge-guided dual-stage parameter-efficient training strategy. This framework not only eliminates the need for costly additional pre-training but also flexibly accommodates various parameter-efficient tuning methods.