# Learning a Unified Embedding for Visual Search at Pinterest

arXiv: https://arxiv.org/abs/1908.01707 · HTML: https://mlsanity.com/paper/1908.01707/

## Key facts

- Cited by 25 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 5 Aug 2019.

## Built on by (most cited first, 25 of 25)

- [Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representations](https://mlsanity.com/paper/2309.01858/) (2023)
- [Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations](https://mlsanity.com/paper/2108.05887/) (2021)
- [ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest](https://mlsanity.com/paper/2205.11728/) (2022)
- [Table Foundation Models: on knowledge pre-training for tabular learning](https://mlsanity.com/paper/2505.14415/) (2025)
- [MIEB: Massive Image Embedding Benchmark](https://mlsanity.com/paper/2504.10471/) (2025)
- [Bringing Multimodality to Amazon Visual Search System](https://mlsanity.com/paper/2412.13364/) (2024)
- [OpenFashionCLIP: Vision-and-Language Contrastive Learning with Open-Source Fashion Data](https://mlsanity.com/paper/2309.05551/) (2023)
- [UDON: Universal Dynamic Online distillatioN for generic image representations](https://mlsanity.com/paper/2406.08332/) (2024)
- [Improving Pinterest Search Relevance Using Large Language Models](https://mlsanity.com/paper/2410.17152/) (2024)
- [adSformers: Personalization from Short-Term Sequences and Diversity of Representations in Etsy Ads](https://mlsanity.com/paper/2302.01255/) (2023)
- [LRVS-Fashion: Extending Visual Search with Referring Instructions](https://mlsanity.com/paper/2306.02928/) (2023)
- [On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction](https://mlsanity.com/paper/2504.08169/) (2025)
- [InteractRank: Personalized Web-Scale Search Pre-Ranking with Cross Interaction Features](https://mlsanity.com/paper/2504.06609/) (2025)
- [OneVision: An End-to-End Generative Framework for Multi-view E-commerce Vision Search](https://mlsanity.com/paper/2510.05759/) (2025)
- [Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models](https://mlsanity.com/paper/2510.13359/) (2025)
- [A machine learning pipeline for aiding school identification from child trafficking images](https://mlsanity.com/paper/2106.05215/) (2021)
- [Visually Similar Products Retrieval for Shopsy](https://mlsanity.com/paper/2210.04560/) (2022)
- [Efficient Large-Scale Visual Representation Learning And Evaluation](https://mlsanity.com/paper/2305.13399/) (2023)
- [Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey](https://mlsanity.com/paper/2308.08234/) (2023)
- [Save, Revisit, Retain: A Scalable Framework for Enhancing User Retention in Large-Scale Recommender Systems](https://mlsanity.com/paper/2511.18013/) (2025)
- [PinEqualizer: Full Funnel Content Exploration and Debiasing System at Pinterest](https://mlsanity.com/paper/2607.22518/) (2026)
- [Visual Product Graph: Bridging Visual Products And Composite Images For End-to-End Style Recommendations](https://mlsanity.com/paper/2505.21454/) (2025)
- [From Pixels to Purchase: Building and Evaluating a Taxonomy-Decoupled Visual Search Engine for Home Goods E-commerce](https://mlsanity.com/paper/2601.11769/) (2026)
- [PinCLIP: Large-scale Foundational Multimodal Representation at Pinterest](https://mlsanity.com/paper/2603.03544/) (2026)
- [Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation](https://mlsanity.com/paper/2607.17499/) (2026)

## Abstract (arXiv)

At Pinterest, we utilize image embeddings throughout our search and recommendation systems to help our users navigate through visual content by powering experiences like browsing of related content and searching for exact products for shopping. In this work we describe a multi-task deep metric learning system to learn a single unified image embedding which can be used to power our multiple visual search products. The solution we present not only allows us to train for multiple application objectives in a single deep neural network architecture, but takes advantage of correlated information in the combination of all training data from each application to generate a unified embedding that outperforms all specialized embeddings previously deployed for each product. We discuss the challenges of handling images from different domains such as camera photos, high quality web images, and clean product catalog images. We also detail how to jointly train for multiple product objectives and how to leverage both engagement data and human labeled data. In addition, our trained embeddings can also be binarized for efficient storage and retrieval without compromising precision and recall. Through comprehensive evaluations on offline metrics, user studies, and online A/B experiments, we demonstrate that our proposed unified embedding improves both relevance and engagement of our visual search products for both browsing and searching purposes when compared to existing specialized embeddings. Finally, the deployment of the unified embedding at Pinterest has drastically reduced the operation and engineering cost of maintaining multiple embeddings while improving quality.