mlsanityData to 14 Oct 2026

NoteLLM-2: Multimodal Large Representation Models for Recommendation

arXiv 2405.16789 ↗First posted 27 May 2024cs.IRcited by 44

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. However, their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. While leveraging existing Multimodal Large Language Models (MLLMs) for such tasks is promising, challenges arise due to their delayed release compared to corresponding LLMs and the inefficiency in representation tasks. To address these issues, we propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation. Preliminary experiments revealed that fine-tuned LLMs often neglect image content. To counteract this, we propose NoteLLM-2, a novel framework that enhances visual information. Specifically, we propose two approaches: first, a prompt-based method that segregates visual and textual content, employing a multimodal In-Context Learning strategy to balance focus across modalities; second, a late fusion technique that directly integrates visual information into the final representations. Extensive experiments, both online and offline, demonstrate the effectiveness of our approach. Code is available at https://github.com/Applied-Machine-Learning-Lab/NoteLLM.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 May 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 26; 2026: 15.

44 papers.

Sort:

By area:

  1. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 35
  2. 2024Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendationcited by 33
  3. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  4. 2025Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifyingcited by 23
  5. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
  6. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systemscited by 19
  7. 2025Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expressioncited by 16
  8. 2025Personalize Before Retrieve: LLM-based Personalized Query Expansion for User-Centric Retrievalcited by 11
  9. 2024Large Language Model Enhanced Recommender Systems: A Surveycited by 10
  10. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  11. 2024Semantic Convergence: Harmonizing Recommender Systems via Two-Stage Alignment and Behavioral Semantic Tokenizationcited by 9
  12. 2025NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendationscited by 6
32 more
  1. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  2. 2025SampleLLM: Optimizing Tabular Data Synthesis in Recommendationscited by 4
  3. 2026Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testingcited by 4
  4. 2025GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networkscited by 3
  5. 2025GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Modelscited by 3
  6. 2025LLM-Aligned Geographic Item Tokenization for Local-Life Recommendationcited by 3
  7. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  8. 2025SUMMA: A Multimodal Large Language Model for Advertisement Summarizationcited by 2
  9. 2025Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selectioncited by 2
  10. 2025Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generationcited by 2
  11. 2026De-conflating Preference and Qualification: Constrained Dual-Perspective Reasoning for Job Recommendation with Large Language Modelscited by 2
  12. 2026IDProxy: CTR Prediction with Multimodal LLMs for Cold-Start Recommendation at Xiaohongshucited by 2
  13. 2026Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memorycited by 2
  14. 2025Distilling Transitional Pattern to Large Language Models for Multimodal Session-based Recommendationcited by 1
  15. 2025Serendipitous Recommendation with Multimodal LLMcited by 1
  16. 2025FOUNDv2: Learning Unified User Quantized Tokenizers for User Representationcited by 1
  17. 2025A Metric for MLLM Alignment in Large-scale Recommendationcited by 1
  18. 2025MIRA: Empowering One-Touch AI Services on Smartphones with MLLM-based Instruction Recommendationcited by 1
  19. 2025Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rayscited by 1
  20. 2025MMSRARec: Summarization and Retrieval Augumented Sequential Recommendation Based on Multimodal Large Language Modelcited by 1
  21. 2026Efficient Sequential Recommendation for Long Term User Interest Via Personalizationcited by 1
  22. 2026MatchLM2Lite: A Scalable MLLM-to-Lite Framework for Reproduced Content Identificationcited by 1
  23. 2025Improving LLM Interpretability and Performance via Guided Embedding Refinement for Sequential Recommendationcited by 0
  24. 2025Boosting Multi-modal Keyphrase Prediction with Dynamic Chain-of-Thought in Vision-Language Modelscited by 0
  25. 2025An item is worth one token in Multimodal Large Language Models-based Sequential Recommendationcited by 0
  26. 2026Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learningcited by 0
  27. 2026VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendationcited by 0
  28. 2026ARGUS: Policy-Adaptive Ad Governance via Evolving Reinforcement with Adversarial Umpiringcited by 0
  29. 2026FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendationcited by 0
  30. 2026UniNote: A Unified Embedding Model for Multimodal Representation and Rankingcited by 0
  31. 2026GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender Systemcited by 0
  32. 2026TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation
  2. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation
  3. 2026A General Framework for Multimodal LLM-Based Multimedia Understanding in Large-Scale Recommendation Systems
  4. 2026VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models
  5. 2025Structural and Disentangled Adaptation of Large Vision Language Models for Multimodal Recommendation
  6. 2024X-Reflect: Cross-Reflection Prompting for Multimodal Recommendation
  7. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models
  8. 2026DMESR: Dual-view MLLM-based Enhancing Framework for Multimodal Sequential Recommendation ↗