# Large Multi-modal Encoders for Recommendation

arXiv: https://arxiv.org/abs/2310.20343 · HTML: https://mlsanity.com/paper/2310.20343/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2023.

## Built on by (most cited first, 5 of 5)

- [Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroup](https://mlsanity.com/paper/2312.05795/) (2023)
- [CFIR: Fast and Effective Long-Text To Image Retrieval for Large Corpora](https://mlsanity.com/paper/2402.15276/) (2024)
- [CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion](https://mlsanity.com/paper/2402.14551/) (2024)
- [LaViC: Adapting Large Vision-Language Models to Visually-Aware Conversational Recommendation](https://mlsanity.com/paper/2503.23312/) (2025)
- [Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learning](https://mlsanity.com/paper/2403.06289/) (2024)

## Abstract (arXiv)

In recent years, the rapid growth of online multimedia services, such as e-commerce platforms, has necessitated the development of personalised recommendation approaches that can encode diverse content about each item. Indeed, modern multi-modal recommender systems exploit diverse features obtained from raw images and item descriptions to enhance the recommendation performance. However, the existing multi-modal recommenders primarily depend on the features extracted individually from different media through pre-trained modality-specific encoders, and exhibit only shallow alignments between different modalities - limiting these systems' ability to capture the underlying relationships between the modalities. In this paper, we investigate the usage of large multi-modal encoders within the specific context of recommender systems, as these have previously demonstrated state-of-the-art effectiveness when ranking items across various domains. Specifically, we tailor two state-of-the-art multi-modal encoders (CLIP and VLMo) for recommendation tasks using a range of strategies, including the exploration of pre-trained and fine-tuned encoders, as well as the assessment of the end-to-end training of these encoders. We demonstrate that pre-trained large multi-modal encoders can generate more aligned and effective user/item representations compared to existing modality-specific encoders across three multi-modal recommendation datasets. Furthermore, we show that fine-tuning these large multi-modal encoders with recommendation datasets leads to an enhanced recommendation performance. In terms of different training paradigms, our experiments highlight the essential role of the end-to-end training of large multi-modal encoders in multi-modal recommendation systems.