# Visual Search at Alibaba

arXiv: https://arxiv.org/abs/2102.04674 · HTML: https://mlsanity.com/paper/2102.04674/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning](https://mlsanity.com/paper/1503.00949/) (2015)
- [Visual Search at eBay](https://mlsanity.com/paper/1706.03154/) (2017)
- [Visual Search at Pinterest](https://mlsanity.com/paper/1505.07647/) (2015)
- [Exploiting Local Features from Deep Networks for Image Retrieval](https://mlsanity.com/paper/1504.05133/) (2015)
- [Self-taught Object Localization with Deep Networks](https://mlsanity.com/paper/1409.3964/) (2014)
- [Neural Codes for Image Retrieval](https://mlsanity.com/paper/1404.1777/) (2014)

## Built on by (most cited first, 27 of 27)

- [From Region to Patch: Attribute-Aware Foreground-Background Contrastive Learning for Fine-Grained Fashion Retrieval](https://mlsanity.com/paper/2305.10260/) (2023)
- [OpenFashionCLIP: Vision-and-Language Contrastive Learning with Open-Source Fashion Data](https://mlsanity.com/paper/2309.05551/) (2023)
- [IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing](https://mlsanity.com/paper/2410.16977/) (2024)
- [I Want This Product but Different : Multimodal Retrieval with Synthetic Query Expansion](https://mlsanity.com/paper/2102.08871/) (2021)
- [Towards Zero-shot Cross-lingual Image Retrieval and Tagging](https://mlsanity.com/paper/2109.07622/) (2021)
- [LRVS-Fashion: Extending Visual Search with Referring Instructions](https://mlsanity.com/paper/2306.02928/) (2023)
- [Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions](https://mlsanity.com/paper/2503.00501/) (2025)
- [LEANN: A Low-Storage Vector Index](https://mlsanity.com/paper/2506.08276/) (2025)
- [REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing](https://mlsanity.com/paper/2506.16444/) (2025)
- [Scalable Disk-Based Approximate Nearest Neighbor Search with Page-Aligned Graph](https://mlsanity.com/paper/2509.25487/) (2025)
- [OneVision: An End-to-End Generative Framework for Multi-view E-commerce Vision Search](https://mlsanity.com/paper/2510.05759/) (2025)
- [Visually Similar Products Retrieval for Shopsy](https://mlsanity.com/paper/2210.04560/) (2022)
- [Efficient Large-Scale Visual Representation Learning And Evaluation](https://mlsanity.com/paper/2305.13399/) (2023)
- [Learning with Multi-modal Gradient Attention for Explainable Composed Image Retrieval](https://mlsanity.com/paper/2308.16649/) (2023)
- [Learning Unified Distance Metric Across Diverse Data Distributions with Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2309.08944/) (2023)
- [Transformer-empowered Multi-modal Item Embedding for Enhanced Image Search in E-Commerce](https://mlsanity.com/paper/2311.17954/) (2023)
- [FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation](https://mlsanity.com/paper/2404.08582/) (2024)
- [Efficient and Discriminative Image Feature Extraction for Universal Image Retrieval](https://mlsanity.com/paper/2409.13513/) (2024)
- [WebANNS: Fast and Efficient Approximate Nearest Neighbor Search in Web Browsers](https://mlsanity.com/paper/2507.00521/) (2025)
- [REVISION:Reflective Intent Mining and Online Reasoning Auxiliary for E-commerce Visual Search System Optimization](https://mlsanity.com/paper/2510.22739/) (2025)
- [Integrating Semantics and Neighborhood Information with Graph-Driven Generative Models for Document Retrieval](https://mlsanity.com/paper/2105.13066/) (2021)
- [Web-Scale Generic Object Detection at Microsoft Bing](https://mlsanity.com/paper/2107.01814/) (2021)
- [Technologies for AI-Driven Fashion Social Networking Service with E-Commerce](https://mlsanity.com/paper/2203.10996/) (2022)
- [Scaling Cross-Domain Content-Based Image Retrieval for E-commerce Snap and Search Application](https://mlsanity.com/paper/2204.11593/) (2022)
- [From Pixels to Purchase: Building and Evaluating a Taxonomy-Decoupled Visual Search Engine for Home Goods E-commerce](https://mlsanity.com/paper/2601.11769/) (2026)
- [ANNLib: A Development Framework for Efficient Approximate Nearest Neighbor Search](https://mlsanity.com/paper/2607.17582/) (2026)
- [RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models](https://mlsanity.com/paper/2609.16847/) (2026)

## Abstract (arXiv)

This paper introduces the large scale visual search algorithm and system infrastructure at Alibaba. The following challenges are discussed under the E-commercial circumstance at Alibaba (a) how to handle heterogeneous image data and bridge the gap between real-shot images from user query and the online images. (b) how to deal with large scale indexing for massive updating data. (c) how to train deep models for effective feature representation without huge human annotations. (d) how to improve the user engagement by considering the quality of the content. We take advantage of large image collection of Alibaba and state-of-the-art deep learning techniques to perform visual search at scale. We present solutions and implementation details to overcome those problems and also share our learnings from building such a large scale commercial visual search engine. Specifically, model and search-based fusion approach is introduced to effectively predict categories. Also, we propose a deep CNN model for joint detection and feature learning by mining user click behavior. The binary index engine is designed to scale up indexing without compromising recall and precision. Finally, we apply all the stages into an end-to-end system architecture, which can simultaneously achieve highly efficient and scalable performance adapting to real-shot images. Extensive experiments demonstrate the advancement of each module in our system. We hope visual search at Alibaba becomes more widely incorporated into today's commercial applications.