# Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace

arXiv: https://arxiv.org/abs/2508.05661 · HTML: https://mlsanity.com/paper/2508.05661/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 5 of 5)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Search at eBay](https://mlsanity.com/paper/1706.03154/) (2017)
- [Extending CLIP for Category-to-image Retrieval in E-commerce](https://mlsanity.com/paper/2112.11294/) (2021)
- [Shop The Look: Building a Large Scale Visual Shopping System at Pinterest](https://mlsanity.com/paper/2006.10866/) (2020)
- [e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce](https://mlsanity.com/paper/2207.00208/) (2022)

## Abstract (arXiv)

Visual search offers an intuitive way for customers to explore diverse product catalogs, particularly in consumer-to-consumer (C2C) marketplaces where listings are often unstructured and visually driven. This paper presents a scalable visual search system deployed in Mercari's C2C marketplace, where end-users act as buyers and sellers. We evaluate recent vision-language models for zero-shot image retrieval and compare their performance with an existing fine-tuned baseline. The system integrates real-time inference and background indexing workflows, supported by a unified embedding pipeline optimized through dimensionality reduction. Offline evaluation using user interaction logs shows that the multilingual SigLIP model outperforms other models across multiple retrieval metrics, achieving a 13.3% increase in nDCG@5 over the baseline. A one-week online A/B test in production further confirms real-world impact, with the treatment group showing substantial gains in engagement and conversion, up to a 40.9% increase in transaction rate via image search. Our findings highlight that recent zero-shot models can serve as a strong and practical baseline for production use, which enables teams to deploy effective visual search systems with minimal overhead, while retaining the flexibility to fine-tune based on future data or domain-specific needs.