mlsanityData to 7 Oct 2026

VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

arXiv 2603.17450 ↗First posted 18 Mar 2026cs.IR cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal embedders for SR. However, we find that standard contrastive Supervised Fine-Tuning (SFT), used to adapt VLMs for embedding generation and inject CF signals, can amplify inherent modality imbalance: optimization becomes dominated by one modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to promote balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to mitigate gradient imbalance during optimization and cross-modal Relational Topology Regularization to preserve geometric consistency between modalities. Experiments show that VLM2Rec consistently improves over strong baselines in both accuracy and robustness across diverse scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 75 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

55 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation19 Aug

Other references · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2018Self-Attentive Sequential Recommendationcited by 1,169
  11. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  12. 2017A Closer Look at Memorization in Deep Networkscited by 936
38 more
  1. 2015Image-based Recommendations on Styles and Substitutescited by 702
  2. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  4. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  5. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  6. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  7. 2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
  8. 2023Improving Text Embeddings with Large Language Modelscited by 295
  9. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  10. 2018An efficient framework for learning sentence representationscited by 197
  11. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  12. 2023Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisitedcited by 124
  13. 2021Contrastive Learning for Representation Degeneration Problem in Sequential Recommendationcited by 121
  14. 2022Towards Universal Sequence Representation Learning for Recommender Systemscited by 120
  15. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  16. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  17. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  18. 2023DiffuRec: A Diffusion Model for Sequential Recommendationcited by 91
  19. 2022Intent Contrastive Learning for Sequential Recommendationcited by 90
  20. 2024Making Text Embedders Few-Shot Learnerscited by 78
  21. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  22. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  23. 2024NoteLLM-2: Multimodal Large Representation Models for Recommendationcited by 44
  24. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  25. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  26. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  27. 2023MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendationcited by 30
  28. 2020Sequential Recommendation with Self-Attentive Multi-Adversarial Networkcited by 26
  29. 2024Can Small Language Models be Good Reasoners for Sequential Recommendation?cited by 25
  30. 2022On Modality Bias Recognition and Reductioncited by 20
  31. 2025LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendationcited by 19
  32. 2024LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendationcited by 18
  33. 2025Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendationcited by 14
  34. 2023Adaptive Multi-Modalities Fusion in Sequential Recommendation Systemscited by 13
  35. 2025Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrievalcited by 12
  36. 2024Sequence-level Semantic Representation Fusion for Recommender Systemscited by 10
  37. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compressioncited by 10
  38. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DMESR: Dual-view MLLM-based Enhancing Framework for Multimodal Sequential Recommendation ↗
  2. 2025An item is worth one token in Multimodal Large Language Models-based Sequential Recommendation
  3. 2025Structural and Disentangled Adaptation of Large Vision Language Models for Multimodal Recommendation
  4. 2026VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models
  5. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation
  6. 2024Molar: Multimodal LLMs with Collaborative Filtering Alignment for Enhanced Sequential Recommendation
  7. 2025MMSRARec: Summarization and Retrieval Augumented Sequential Recommendation Based on Multimodal Large Language Model
  8. 2025Bridging Domain Gaps between Pretrained Multimodal Models and Recommendations