mlsanityData to 7 Oct 2026

Delving into E-Commerce Product Retrieval with Vision-Language Pre-training

arXiv 2304.04377 ↗First posted 10 Apr 2023cs.IRcited by 2

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

E-commerce search engines comprise a retrieval phase and a ranking phase, where the first one returns a candidate product set given user queries. Recently, vision-language pre-training, combining textual information with visual clues, has been popular in the application of retrieval tasks. In this paper, we propose a novel V+L pre-training method to solve the retrieval problem in Taobao Search. We design a visual pre-training task based on contrastive learning, outperforming common regression-based visual pre-training tasks. In addition, we adopt two negative sampling schemes, tailored for the large-scale retrieval task. Besides, we introduce the details of the online deployment of our proposed method in real-world situations. Extensive offline/online experiments demonstrate the superior performance of our method on the retrieval task. Our proposed method is employed as one retrieval channel of Taobao Search and serves hundreds of millions of users in real time.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 18 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

16 of 18 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  9. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  10. 2020FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrievalcited by 55
  11. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  12. 2021Pre-trained Language Model based Ranking in Baidu Searchcited by 33
4 more
  1. 2022CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrievalcited by 25
  2. 2021Cross-Batch Negative Sampling for Training Two-Tower Recommenderscited by 24
  3. 2021Learning a Product Relevance Model from Click-Through Data in E-Commercecited by 22
  4. 2020EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applicationscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understandingcited by 5
  2. 2025Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search
  2. 2022V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval
  3. 2023Query-LIFE: Query-aware Language Image Fusion Embedding for E-Commerce Relevance
  4. 2026Semantic Retrieval for Product Search in E-Commerce
  5. 2025Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search
  6. 2025Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Search
  7. 2025TaoSR1: The Thinking Model for E-commerce Relevance Search
  8. 2025Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models