mlsanityData to 7 Oct 2026

Extending CLIP for Category-to-image Retrieval in E-commerce

arXiv 2112.11294 ↗First posted 21 Dec 2021cs.IR cs.LG cs.MMcited by 24

Not on the map. Why?

Abstract (arXiv)

E-commerce provides rich multimodal data that is barely leveraged in practice. One aspect of this data is a category tree that is being used in search and recommendation. However, in practice, during a user's session there is often a mismatch between a textual and a visual representation of a given category. Motivated by the problem, we introduce the task of category-to-image retrieval in e-commerce and propose a model for the task, CLIP-ITA. The model leverages information from multiple modalities (textual, visual, and attribute modality) to create product representations. We explore how adding information from multiple modalities (textual, visual, and attribute modality) impacts the model's performance. In particular, we observe that CLIP-ITA significantly outperforms a comparable model that leverages only the visual modality and a comparable model that leverages the visual and attribute modality.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 21 Dec 2021.

Builds on

22 of 44 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2016Layer Normalizationcited by 5,579
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  10. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  11. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  12. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
10 more
  1. 2015Natural Language Object Retrievalcited by 124
  2. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  3. 2020Hierarchical Fashion Graph Network for Personalized Outfit Recommendationcited by 27
  4. 2018DeepStyle: Multimodal Search Engine for Fashion and Interior Designcited by 14
  5. 2019Improving Outfit Recommendation with Co-supervision of Fashion Generationcited by 6
  6. 2020How to Grow a (Product) Tree: Personalized Category Suggestions for eCommerce Type-Aheadcited by 6
  7. 2019Multi-Label Product Categorization Using Multi-Modal Fusion Modelscited by 3
  8. 2018One-Shot Item Search with Multimodal Datacited by 1
  9. 2019The Resale Price Prediction of Secondhand Jewelry Items Using a Multi-modal Deep Model with Iterative Co-Attentioncited by 1
  10. 2021Category Aware Explainable Conversational Recommendationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 5; 2025: 8; 2026: 1.

24 papers.

Sort:

By area:

  1. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  2. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  3. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
  4. 2023Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Imagescited by 19
  5. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  6. 2025MADGEN: Mass-Spec attends to De Novo Molecular generationcited by 16
  7. 2024Multimodal Learned Sparse Retrieval with Probabilistic Expansion Controlcited by 12
  8. 2024Benchmark Granularity and Model Robustness for Image-Text Retrievalcited by 5
  9. 2023Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Studycited by 3
  10. 2023Self Contrastive Learning for Session-based Recommendationcited by 3
  11. 2023Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysiscited by 2
  12. 2025Prototype Perturbation for Relaxing Alignment Constraints in Backward-Compatible Learningcited by 2
12 more
  1. 2022Unimodal vs. Multimodal Siamese Networks for Outfit Completioncited by 1
  2. 2023Towards Contrastive Learning in Music Video Domaincited by 1
  3. 2025Adapting Vision-Language Models for Evaluating World Modelscited by 1
  4. 2025LLM4Rec: Large Language Models for Multimodal Generative Recommendation with Causal Debiasingcited by 1
  5. 2026Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understandingcited by 1
  6. 2023Lexical Entrainment for Conversational Systemscited by 0
  7. 2024CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarioscited by 0
  8. 2024Multi-method Integration with Confidence-based Weighting for Zero-shot Image Classificationcited by 0
  9. 2025Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gapcited by 0
  10. 2025Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplacecited by 0
  11. 2025DesignCLIP: Multimodal Learning with CLIP for Design Patent Understandingcited by 0
  12. 2025Image-Seeking Intent Prediction for Cross-Device Product Searchcited by 0