# Shop The Look: Building a Large Scale Visual Shopping System at Pinterest

arXiv: https://arxiv.org/abs/2006.10866 · HTML: https://mlsanity.com/paper/2006.10866/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 12 of 12)

- [Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations](https://mlsanity.com/paper/2108.05887/) (2021)
- [e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce](https://mlsanity.com/paper/2207.00208/) (2022)
- [OpenFashionCLIP: Vision-and-Language Contrastive Learning with Open-Source Fashion Data](https://mlsanity.com/paper/2309.05551/) (2023)
- [Sublinear Time Algorithm for Online Weighted Bipartite Matching](https://mlsanity.com/paper/2208.03367/) (2022)
- [LRVS-Fashion: Extending Visual Search with Referring Instructions](https://mlsanity.com/paper/2306.02928/) (2023)
- [OneVision: An End-to-End Generative Framework for Multi-view E-commerce Vision Search](https://mlsanity.com/paper/2510.05759/) (2025)
- [FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation](https://mlsanity.com/paper/2404.08582/) (2024)
- [Web-Scale Generic Object Detection at Microsoft Bing](https://mlsanity.com/paper/2107.01814/) (2021)
- [Visual Product Graph: Bridging Visual Products And Composite Images For End-to-End Style Recommendations](https://mlsanity.com/paper/2505.21454/) (2025)
- [Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace](https://mlsanity.com/paper/2508.05661/) (2025)
- [Optimizing Product Deduplication in E-Commerce with Multimodal Embeddings](https://mlsanity.com/paper/2509.15858/) (2025)
- [LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data](https://mlsanity.com/paper/2607.28880/) (2026)

## Abstract (arXiv)

As online content becomes ever more visual, the demand for searching by visual queries grows correspondingly stronger. Shop The Look is an online shopping discovery service at Pinterest, leveraging visual search to enable users to find and buy products within an image. In this work, we provide a holistic view of how we built Shop The Look, a shopping oriented visual search system, along with lessons learned from addressing shopping needs. We discuss topics including core technology across object detection and visual embeddings, serving infrastructure for realtime inference, and data labeling methodology for training/evaluation data collection and human evaluation. The user-facing impacts of our system design choices are measured through offline evaluations, human relevance judgements, and online A/B experiments. The collective improvements amount to cumulative relative gains of over 160% in end-to-end human relevance judgements and over 80% in engagement. Shop The Look is deployed in production at Pinterest.