# VisualLens: Personalization through Task-Agnostic Visual History

arXiv: https://arxiv.org/abs/2411.16034 · HTML: https://mlsanity.com/paper/2411.16034/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback](https://mlsanity.com/paper/1510.01784/) (2015)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2308.11131/) (2023)
- [Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences](https://mlsanity.com/paper/2307.14225/) (2023)
- [Collaborative Large Language Model for Recommender Systems](https://mlsanity.com/paper/2311.01343/) (2023)
- [GenRec: Large Language Model for Generative Recommendation](https://mlsanity.com/paper/2307.00457/) (2023)
- [ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models](https://mlsanity.com/paper/2305.06566/) (2023)
- [Teach LLMs to Personalize -- An Approach inspired by Writing Education](https://mlsanity.com/paper/2308.07968/) (2023)
- [Multi-Modal Self-Supervised Learning for Recommendation](https://mlsanity.com/paper/2302.10632/) (2023)
- [Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendation](https://mlsanity.com/paper/2207.05584/) (2022)
- [IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations](https://mlsanity.com/paper/2404.01266/) (2024)
- [UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining](https://mlsanity.com/paper/2202.13469/) (2022)
- [Personalized Showcases: Generating Multi-Modal Explanations for Recommendations](https://mlsanity.com/paper/2207.00422/) (2022)
- [Adversarial Training Towards Robust Multimedia Recommender System](https://mlsanity.com/paper/1809.07062/) (2018)
- [TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback](https://mlsanity.com/paper/2206.06190/) (2022)
- [Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond](https://mlsanity.com/paper/2403.10667/) (2024)
- [Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?](https://mlsanity.com/paper/2404.18624/) (2024)

## Abstract (arXiv)

Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals. However, item-based histories are not always accessible, and are not generalizable for multimodal recommendation. We hypothesize that a user's visual history -- comprising images from daily life -- can offer rich, task-agnostic insights into their interests and preferences, and thus be leveraged for effective personalization. To this end, we propose VisualLens, a novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task-agnostic visual history. VisualLens extracts, filters, and refines a spectrum user profile from the visual history to support personalized recommendation. We created two new benchmarks, Google-Review-V and Yelp-V, with task-agnostic visual histories, and show that VisualLens improves over state-of-the-art item-based multimodal recommendations by 5-10% on Hit@3, and outperforms GPT-4o by 2-5%. Further analysis shows that VisualLens is robust across varying history lengths and excels at adapting to both longer histories and unseen content categories.