# NoteLLM-2: Multimodal Large Representation Models for Recommendation

arXiv: https://arxiv.org/abs/2405.16789 · HTML: https://mlsanity.com/paper/2405.16789/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2024.

## Built on by (most cited first, 30 of 44)

- [MemVerse: Multimodal Memory for Lifelong Learning Agents](https://mlsanity.com/paper/2512.03627/) (2025)
- [Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2408.09698/) (2024)
- [CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning](https://mlsanity.com/paper/2503.19900/) (2025)
- [Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying](https://mlsanity.com/paper/2506.02020/) (2025)
- [Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization](https://mlsanity.com/paper/2504.18397/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression](https://mlsanity.com/paper/2505.13527/) (2025)
- [Personalize Before Retrieve: LLM-based Personalized Query Expansion for User-Centric Retrieval](https://mlsanity.com/paper/2510.08935/) (2025)
- [Large Language Model Enhanced Recommender Systems: A Survey](https://mlsanity.com/paper/2412.13432/) (2024)
- [SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model](https://mlsanity.com/paper/2510.12709/) (2025)
- [Semantic Convergence: Harmonizing Recommender Systems via Two-Stage Alignment and Behavioral Semantic Tokenization](https://mlsanity.com/paper/2412.13771/) (2024)
- [NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations](https://mlsanity.com/paper/2511.18793/) (2025)
- [TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings](https://mlsanity.com/paper/2603.04772/) (2026)
- [SampleLLM: Optimizing Tabular Data Synthesis in Recommendations](https://mlsanity.com/paper/2501.16125/) (2025)
- [Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing](https://mlsanity.com/paper/2601.04554/) (2026)
- [GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks](https://mlsanity.com/paper/2506.16114/) (2025)
- [GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Models](https://mlsanity.com/paper/2507.06507/) (2025)
- [LLM-Aligned Geographic Item Tokenization for Local-Life Recommendation](https://mlsanity.com/paper/2511.14221/) (2025)
- [Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs](https://mlsanity.com/paper/2602.05275/) (2026)
- [SUMMA: A Multimodal Large Language Model for Advertisement Summarization](https://mlsanity.com/paper/2508.20582/) (2025)
- [Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection](https://mlsanity.com/paper/2511.17587/) (2025)
- [Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.19257/) (2025)
- [De-conflating Preference and Qualification: Constrained Dual-Perspective Reasoning for Job Recommendation with Large Language Models](https://mlsanity.com/paper/2602.03097/) (2026)
- [IDProxy: CTR Prediction with Multimodal LLMs for Cold-Start Recommendation at Xiaohongshu](https://mlsanity.com/paper/2603.01590/) (2026)
- [Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory](https://mlsanity.com/paper/2605.00702/) (2026)
- [Distilling Transitional Pattern to Large Language Models for Multimodal Session-based Recommendation](https://mlsanity.com/paper/2504.10538/) (2025)
- [Serendipitous Recommendation with Multimodal LLM](https://mlsanity.com/paper/2506.08283/) (2025)
- [FOUNDv2: Learning Unified User Quantized Tokenizers for User Representation](https://mlsanity.com/paper/2508.00956/) (2025)
- [A Metric for MLLM Alignment in Large-scale Recommendation](https://mlsanity.com/paper/2508.04963/) (2025)
- [MIRA: Empowering One-Touch AI Services on Smartphones with MLLM-based Instruction Recommendation](https://mlsanity.com/paper/2509.13773/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. However, their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. While leveraging existing Multimodal Large Language Models (MLLMs) for such tasks is promising, challenges arise due to their delayed release compared to corresponding LLMs and the inefficiency in representation tasks. To address these issues, we propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation. Preliminary experiments revealed that fine-tuned LLMs often neglect image content. To counteract this, we propose NoteLLM-2, a novel framework that enhances visual information. Specifically, we propose two approaches: first, a prompt-based method that segregates visual and textual content, employing a multimodal In-Context Learning strategy to balance focus across modalities; second, a late fusion technique that directly integrates visual information into the final representations. Extensive experiments, both online and offline, demonstrate the effectiveness of our approach. Code is available at https://github.com/Applied-Machine-Learning-Lab/NoteLLM.