PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest

arXiv 2603.03544 ↗First posted 3 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

While multi-modal Visual Language Models (VLMs) have demonstrated significant success across various domains, the integration of VLMs into recommendation and retrieval systems remains a challenge, due to issues like training objective discrepancies and serving efficiency bottlenecks. This paper introduces PinCLIP, a large-scale visual representation learning approach developed to enhance retrieval and ranking models at Pinterest by leveraging VLMs to learn image-text alignment. We propose a novel hybrid Vision Transformer architecture that utilizes a VLM backbone and a hybrid fusion mechanism to capture multi-modality content representation at varying granularities. Beyond standard image-to-text alignment objectives, we introduce a neighbor alignment objective to model the cross-fusion of multi-modal representations within the Pinterest Pin-Board graph. Offline evaluations show that PinCLIP outperforms state-of-the-art baselines, such as Qwen, by 20% in multi-modal retrieval tasks. Online A/B testing demonstrates significant business impact, including substantial engagement gains across all major surfaces in Pinterest. Notably, PinCLIP significantly addresses the "cold-start" problem, enhancing fresh content distribution with a 15% Repin increase in organic content and 8.7% higher click for new Ads.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 57 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

42 of 57 references.

References · 42

2017Attention Is All You Needcited by 59,932
2015Deep Residual Learning for Image Recognitioncited by 51,809
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551

30 more

2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
2018Self-Attentive Sequential Recommendationcited by 1,169
2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
2021Early Convolutions Help Transformers See Bettercited by 395
2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 213
2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
2023Representation Learning with Large Language Models for Recommendationcited by 155
2023Text Is All You Need: Learning Language Representations for Sequential Recommendationcited by 149
2017Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Timecited by 46
2021Zero-Shot Recommender Systemscited by 45
2020What does BERT know about books, movies and music? Probing BERT for Conversational Recommendationcited by 36
2021Improving Conversational Recommendation Systems' Quality with Context-Aware Item Meta Informationcited by 26
2024IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFTcited by 26
2019Learning a Unified Embedding for Visual Search at Pinterestcited by 25
2021Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representationscited by 24
2024CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captionscited by 22
2025Grounding Multilingual Multimodal LLMs With Cultural Knowledgecited by 16
2024OmniSearchSage: Multi-Task Multi-Entity Embeddings for Pinterest Searchcited by 13
2025Revisiting Funnel Transformers for Modern LLM Architectures with Comprehensive Ablations in Training and Inference Configurationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way