# Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation

arXiv: https://arxiv.org/abs/2508.04571 · HTML: https://mlsanity.com/paper/2508.04571/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Efficient Multimodal Learning from Data-centric Perspective](https://mlsanity.com/paper/2402.11530/) (2024)
- [Mining Latent Structures for Multimedia Recommendation](https://mlsanity.com/paper/2104.09036/) (2021)
- [Bootstrap Latent Representations for Multi-modal Recommendation](https://mlsanity.com/paper/2207.05969/) (2022)
- [A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation](https://mlsanity.com/paper/2211.06924/) (2022)
- [Bidirectional Language Models Are Also Few-shot Learners](https://mlsanity.com/paper/2209.14500/) (2022)
- [Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation](https://mlsanity.com/paper/2103.02590/) (2021)
- [LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model](https://mlsanity.com/paper/2404.01331/) (2024)
- [MEGCF: Multimodal Entity Graph Collaborative Filtering for Personalized Recommendation](https://mlsanity.com/paper/2210.07443/) (2022)
- [Formalizing Multimedia Recommendation through Multimodal Deep Learning](https://mlsanity.com/paper/2309.05273/) (2023)
- [A Deep Multimodal Approach for Cold-start Music Recommendation](https://mlsanity.com/paper/1706.09739/) (2017)
- [Ducho 2.0: Towards a More Up-to-Date Unified Framework for the Extraction of Multimodal Features in Recommendation](https://mlsanity.com/paper/2403.04503/) (2024)
- [Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation](https://mlsanity.com/paper/2501.07110/) (2025)
- [Large-scale Benchmarks for Multimodal Recommendation with Ducho](https://mlsanity.com/paper/2409.15857/) (2024)

## Built on by (most cited first, 3 of 3)

- [RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment](https://mlsanity.com/paper/2602.00682/) (2026)
- [VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2603.17450/) (2026)
- [Multimodal Music Recommendation System using LLMs](https://mlsanity.com/paper/2606.00125/) (2026)

## Abstract (arXiv)

Multimodal Recommender Systems aim to improve recommendation accuracy by integrating heterogeneous content, such as images and textual metadata. While effective, it remains unclear whether their gains stem from true multimodal understanding or increased model complexity. This work investigates the role of multimodal item embeddings, emphasizing the semantic informativeness of the representations. Initial experiments reveal that embeddings from standard extractors (e.g., ResNet50, Sentence-Bert) enhance performance, but rely on modality-specific encoders and ad hoc fusion strategies that lack control over cross-modal alignment. To overcome these limitations, we leverage Large Vision-Language Models (LVLMs) to generate multimodal-by-design embeddings via structured prompts. This approach yields semantically aligned representations without requiring any fusion. Experiments across multiple settings show notable performance improvements. Furthermore, LVLMs embeddings offer a distinctive advantage: they can be decoded into structured textual descriptions, enabling direct assessment of their multimodal comprehension. When such descriptions are incorporated as side content into recommender systems, they improve recommendation performance, empirically validating the semantic depth and alignment encoded within LVLMs outputs. Our study highlights the importance of semantically rich representations and positions LVLMs as a compelling foundation for building robust and meaningful multimodal representations in recommendation tasks.