# Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining

arXiv: https://arxiv.org/abs/2107.14572 · HTML: https://mlsanity.com/paper/2107.14572/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [PCL: Proposal Cluster Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1807.03342/) (2018)
- [A Comprehensive Survey on Cross-modal Retrieval](https://mlsanity.com/paper/1607.06215/) (2016)
- [Visually-Aware Fashion Recommendation and Design with Generative Image Models](https://mlsanity.com/paper/1711.02231/) (2017)
- [Medical Image Retrieval using Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.08472/) (2017)
- [Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection](https://mlsanity.com/paper/1907.10164/) (2019)
- [Fashion Retrieval via Graph Reasoning Networks on a Similarity Pyramid](https://mlsanity.com/paper/1908.11754/) (2019)
- [RPC: A Large-Scale Retail Product Checkout Dataset](https://mlsanity.com/paper/1901.07249/) (2019)
- [Triplet-Based Deep Hashing Network for Cross-Modal Retrieval](https://mlsanity.com/paper/1904.02449/) (2019)
- [Complete the Look: Scene-based Complementary Product Recommendation](https://mlsanity.com/paper/1812.01748/) (2018)
- [Leveraging Weakly Annotated Data for Fashion Image Retrieval and Label Prediction](https://mlsanity.com/paper/1709.09426/) (2017)
- [Cross-Domain Image Retrieval with Attention Modeling](https://mlsanity.com/paper/1709.01784/) (2017)
- [Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrieval](https://mlsanity.com/paper/1703.06618/) (2017)
- [A Revisit on Deep Hashings for Large-scale Content Based Image Retrieval](https://mlsanity.com/paper/1711.06016/) (2017)

## Built on by (most cited first, 30 of 42)

- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark](https://mlsanity.com/paper/2202.06767/) (2022)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features](https://mlsanity.com/paper/2308.11485/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [ILIAS: Instance-Level Image retrieval At Scale](https://mlsanity.com/paper/2502.11748/) (2025)
- [CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation](https://mlsanity.com/paper/2308.07146/) (2023)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining](https://mlsanity.com/paper/2109.04275/) (2021)
- [Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation](https://mlsanity.com/paper/2312.08078/) (2023)
- [Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.01431/) (2024)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Instance-Level Composed Image Retrieval](https://mlsanity.com/paper/2510.25387/) (2025)
- [Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-commerce](https://mlsanity.com/paper/2304.02853/) (2023)
- [Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmark](https://mlsanity.com/paper/2406.09016/) (2024)
- [CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation](https://mlsanity.com/paper/2407.01081/) (2024)
- [Modality-aware Transformer for Financial Time series Forecasting](https://mlsanity.com/paper/2310.01232/) (2023)
- [MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2508.11999/) (2025)
- [MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2511.12449/) (2025)
- [Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction](https://mlsanity.com/paper/2206.13749/) (2022)
- [LRVS-Fashion: Extending Visual Search with Referring Instructions](https://mlsanity.com/paper/2306.02928/) (2023)
- [Training with Product Digital Twins for AutoRetail Checkout](https://mlsanity.com/paper/2308.09708/) (2023)
- [TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perception](https://mlsanity.com/paper/2511.19509/) (2025)
- [Cross-Domain Product Representation Learning for Rich-Content E-Commerce](https://mlsanity.com/paper/2308.05550/) (2023)
- [Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval](https://mlsanity.com/paper/2206.08842/) (2022)
- [Patch-wise Retrieval: A Bag of Practical Techniques for Instance-level Matching](https://mlsanity.com/paper/2512.12610/) (2025)
- [Self-Contained Entity Discovery from Captioned Videos](https://mlsanity.com/paper/2208.06662/) (2022)
- [UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning](https://mlsanity.com/paper/2306.00813/) (2023)

## Abstract (arXiv)

Nowadays, customer's demands for E-commerce are more diversified, which introduces more complications to the product retrieval industry. Previous methods are either subject to single-modal input or perform supervised image-level product retrieval, thus fail to accommodate real-life scenarios where enormous weakly annotated multi-modal data are present. In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories. To promote the study of this challenging task, we contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. Notably, Product1M contains over 1 million image-caption pairs and consists of two sample types, i.e., single-product and multi-product samples, which encompass a wide variety of cosmetics brands. In addition to the great diversity, Product1M enjoys several appealing characteristics including fine-grained categories, complex combinations, and fuzzy correspondence that well mimic the real-world scenes. Moreover, we propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE), that excels in capturing the potential synergy between multi-modal inputs via a hybrid-stream transformer in a self-supervised manner.CAPTURE generates discriminative instance features via masked multi-modal learning as well as cross-modal contrastive pretraining and it outperforms several SOTA cross-modal baselines. Extensive ablation studies well demonstrate the effectiveness and the generalization capacity of our model. Dataset and codes are available at https: //github.com/zhanxlin/Product1M.