mlsanityData to 7 Oct 2026

A General Framework for Multimodal LLM-Based Multimedia Understanding in Large-Scale Recommendation Systems

arXiv 2605.09338 ↗First posted 10 May 2026cs.IRcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Conventional recommendation systems frequently fail to fully exploit the high-dimensional semantic signals inherent in multimedia content, thereby limiting the fidelity of user preference modeling. While Multimodal Large Language Models (MM-LLMs) offer robust mechanisms for interpreting such complex data, their integration into latency-constrained, industrial-scale architectures remains a significant challenge. To address this, we propose a generalized framework for MM-LLM-driven multimedia understanding. Our methodology employs a tripartite architecture encompassing content interpretation, representation extraction, and systematic pipeline integration, instantiated via a LLaMA2-based model that generates descriptive captions subsequently ingested as tokenized categorical features. Empirical evaluation demonstrates the efficacy of this approach, yielding a 0.35% increase in offline AUC and a 0.02% improvement in online metrics at scale, substantiating the practical viability of leveraging MM-LLMs to enhance large-scale recommendation performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 21 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

6 of 21 references.

References · 6

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  5. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  6. 2017Hash Embeddings for Efficient Word Representationscited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Serendipitous Recommendation with Multimodal LLM
  2. 2024NoteLLM-2: Multimodal Large Representation Models for Recommendation
  3. 2024Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation
  4. 2025MMSRARec: Summarization and Retrieval Augumented Sequential Recommendation Based on Multimodal Large Language Model
  5. 2025Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations
  6. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation
  7. 2025An item is worth one token in Multimodal Large Language Models-based Sequential Recommendation
  8. 2024Molar: Multimodal LLMs with Collaborative Filtering Alignment for Enhanced Sequential Recommendation