# Evolution of a Web-Scale Near Duplicate Image Detection System

arXiv: https://arxiv.org/abs/2209.08433 · HTML: https://mlsanity.com/paper/2209.08433/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 35 references that are in the map.
- First posted on arXiv 18 Sep 2022.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time](https://mlsanity.com/paper/1711.07601/) (2017)
- [Scalable Object Detection using Deep Neural Networks](https://mlsanity.com/paper/1312.2249/) (2013)
- [Visual Search at Pinterest](https://mlsanity.com/paper/1505.07647/) (2015)
- [Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images](https://mlsanity.com/paper/1611.08321/) (2016)

## Built on by (most cited first, 1 of 1)

- [Visual Product Graph: Bridging Visual Products And Composite Images For End-to-End Style Recommendations](https://mlsanity.com/paper/2505.21454/) (2025)

## Abstract (arXiv)

Detecting near duplicate images is fundamental to the content ecosystem of photo sharing web applications. However, such a task is challenging when involving a web-scale image corpus containing billions of images. In this paper, we present an efficient system for detecting near duplicate images across 8 billion images. Our system consists of three stages: candidate generation, candidate selection, and clustering. We also demonstrate that this system can be used to greatly improve the quality of recommendations and search results across a number of real-world applications. In addition, we include the evolution of the system over the course of six years, bringing out experiences and lessons on how new systems are designed to accommodate organic content growth as well as the latest technology. Finally, we are releasing a human-labeled dataset of ~53,000 pairs of images introduced in this paper.