# A Metric for MLLM Alignment in Large-scale Recommendation

arXiv: https://arxiv.org/abs/2508.04963 · HTML: https://mlsanity.com/paper/2508.04963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)
- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023)
- [MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering](https://mlsanity.com/paper/2405.11985/) (2024)
- [MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation](https://mlsanity.com/paper/2503.10497/) (2025)
- [Bootstrap Latent Representations for Multi-modal Recommendation](https://mlsanity.com/paper/2207.05969/) (2022)
- [NoteLLM-2: Multimodal Large Representation Models for Recommendation](https://mlsanity.com/paper/2405.16789/) (2024)
- [A Critical Study on Data Leakage in Recommender System Offline Evaluation](https://mlsanity.com/paper/2010.11060/) (2020)
- [Large Scale Product Graph Construction for Recommendation in E-commerce](https://mlsanity.com/paper/2010.05525/) (2020)
- [Enhancing Taobao Display Advertising with Multimodal Representations: Challenges, Approaches and Insights](https://mlsanity.com/paper/2407.19467/) (2024)
- [Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving](https://mlsanity.com/paper/2409.02914/) (2024)
- [AlignRec: Aligning and Training in Multimodal Recommendations](https://mlsanity.com/paper/2403.12384/) (2024)
- [Sliding Spectrum Decomposition for Diversified Recommendation](https://mlsanity.com/paper/2107.05204/) (2021)
- [Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts](https://mlsanity.com/paper/2308.10410/) (2023)
- [Towards Large-scale Generative Ranking](https://mlsanity.com/paper/2505.04180/) (2025)
- [On the User Behavior Leakage from Recommender System Exposure](https://mlsanity.com/paper/2210.08435/) (2022)
- [End-to-end training of Multimodal Model and ranking Model](https://mlsanity.com/paper/2404.06078/) (2024)
- [User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems](https://mlsanity.com/paper/2504.05522/) (2025)
- [Who You Are Matters: Bridging Topics and Social Roles via LLM-Enhanced Logical Recommendation](https://mlsanity.com/paper/2505.10940/) (2025)
- [ESANS: Effective and Semantic-Aware Negative Sampling for Large-Scale Retrieval Systems](https://mlsanity.com/paper/2502.16077/) (2025)

## Built on by (most cited first, 1 of 1)

- [IDProxy: CTR Prediction with Multimodal LLMs for Cold-Start Recommendation at Xiaohongshu](https://mlsanity.com/paper/2603.01590/) (2026)

## Abstract (arXiv)

Multimodal recommendation has emerged as a critical technique in modern recommender systems, leveraging content representations from advanced multimodal large language models (MLLMs). To ensure these representations are well-adapted, alignment with the recommender system is essential. However, evaluating the alignment of MLLMs for recommendation presents significant challenges due to three key issues: (1) static benchmarks are inaccurate because of the dynamism in real-world applications, (2) evaluations with online system, while accurate, are prohibitively expensive at scale, and (3) conventional metrics fail to provide actionable insights when learned representations underperform. To address these challenges, we propose the Leakage Impact Score (LIS), a novel metric for multimodal recommendation. Rather than directly assessing MLLMs, LIS efficiently measures the upper bound of preference data. We also share practical insights on deploying MLLMs with LIS in real-world scenarios. Online A/B tests on both Content Feed and Display Ads of Xiaohongshu's Explore Feed production demonstrate the effectiveness of our proposed method, showing significant improvements in user spent time and advertiser value.