# Extending CLIP for Category-to-image Retrieval in E-commerce

arXiv: https://arxiv.org/abs/2112.11294 · HTML: https://mlsanity.com/paper/2112.11294/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Contrastive Learning for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/2006.09920/) (2020)
- [Hierarchical Fashion Graph Network for Personalized Outfit Recommendation](https://mlsanity.com/paper/2005.12566/) (2020)
- [DeepStyle: Multimodal Search Engine for Fashion and Interior Design](https://mlsanity.com/paper/1801.03002/) (2018)
- [Improving Outfit Recommendation with Co-supervision of Fashion Generation](https://mlsanity.com/paper/1908.09104/) (2019)
- [How to Grow a (Product) Tree: Personalized Category Suggestions for eCommerce Type-Ahead](https://mlsanity.com/paper/2005.12781/) (2020)
- [Multi-Label Product Categorization Using Multi-Modal Fusion Models](https://mlsanity.com/paper/1907.00420/) (2019)
- [One-Shot Item Search with Multimodal Data](https://mlsanity.com/paper/1811.10969/) (2018)
- [The Resale Price Prediction of Secondhand Jewelry Items Using a Multi-modal Deep Model with Iterative Co-Attention](https://mlsanity.com/paper/1907.00661/) (2019)
- [Category Aware Explainable Conversational Recommendation](https://mlsanity.com/paper/2103.08733/) (2021)

## Built on by (most cited first, 24 of 24)

- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2309.05173/) (2023)
- [LidarCLIP or: How I Learned to Talk to Point Clouds](https://mlsanity.com/paper/2212.06858/) (2022)
- [Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images](https://mlsanity.com/paper/2307.15904/) (2023)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [MADGEN: Mass-Spec attends to De Novo Molecular generation](https://mlsanity.com/paper/2501.01950/) (2025)
- [Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control](https://mlsanity.com/paper/2402.17535/) (2024)
- [Benchmark Granularity and Model Robustness for Image-Text Retrieval](https://mlsanity.com/paper/2407.15239/) (2024)
- [Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study](https://mlsanity.com/paper/2301.05174/) (2023)
- [Self Contrastive Learning for Session-based Recommendation](https://mlsanity.com/paper/2306.01266/) (2023)
- [Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis](https://mlsanity.com/paper/2306.07664/) (2023)
- [Prototype Perturbation for Relaxing Alignment Constraints in Backward-Compatible Learning](https://mlsanity.com/paper/2503.14824/) (2025)
- [Unimodal vs. Multimodal Siamese Networks for Outfit Completion](https://mlsanity.com/paper/2207.10355/) (2022)
- [Towards Contrastive Learning in Music Video Domain](https://mlsanity.com/paper/2309.00347/) (2023)
- [Adapting Vision-Language Models for Evaluating World Models](https://mlsanity.com/paper/2506.17967/) (2025)
- [LLM4Rec: Large Language Models for Multimodal Generative Recommendation with Causal Debiasing](https://mlsanity.com/paper/2510.01622/) (2025)
- [Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding](https://mlsanity.com/paper/2603.29258/) (2026)
- [Lexical Entrainment for Conversational Systems](https://mlsanity.com/paper/2310.09651/) (2023)
- [CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios](https://mlsanity.com/paper/2401.10475/) (2024)
- [Multi-method Integration with Confidence-based Weighting for Zero-shot Image Classification](https://mlsanity.com/paper/2405.02155/) (2024)
- [Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap](https://mlsanity.com/paper/2501.14826/) (2025)
- [Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace](https://mlsanity.com/paper/2508.05661/) (2025)
- [DesignCLIP: Multimodal Learning with CLIP for Design Patent Understanding](https://mlsanity.com/paper/2508.15297/) (2025)
- [Image-Seeking Intent Prediction for Cross-Device Product Search](https://mlsanity.com/paper/2511.14764/) (2025)

## Abstract (arXiv)

E-commerce provides rich multimodal data that is barely leveraged in practice. One aspect of this data is a category tree that is being used in search and recommendation. However, in practice, during a user's session there is often a mismatch between a textual and a visual representation of a given category. Motivated by the problem, we introduce the task of category-to-image retrieval in e-commerce and propose a model for the task, CLIP-ITA. The model leverages information from multiple modalities (textual, visual, and attribute modality) to create product representations. We explore how adding information from multiple modalities (textual, visual, and attribute modality) impacts the model's performance. In particular, we observe that CLIP-ITA significantly outperforms a comparable model that leverages only the visual modality and a comparable model that leverages the visual and attribute modality.