# Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap

arXiv: https://arxiv.org/abs/2501.14826 · HTML: https://mlsanity.com/paper/2501.14826/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [Fashion-Gen: The Generative Fashion Dataset and Challenge](https://mlsanity.com/paper/1806.08317/) (2018)
- [Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval](https://mlsanity.com/paper/2209.00179/) (2022)
- [Semantic Models for the First-stage Retrieval: A Comprehensive Review](https://mlsanity.com/paper/2103.04831/) (2021)
- [Semantic Product Search](https://mlsanity.com/paper/1907.00937/) (2019)
- [Embedding-based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2106.09297/) (2021)
- [Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback](https://mlsanity.com/paper/2108.13454/) (2021)
- [Query expansion techniques for information retrieval: A survey](https://mlsanity.com/paper/1708.00247/) (2017)
- [Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning](https://mlsanity.com/paper/2006.02282/) (2020)
- [Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval](https://mlsanity.com/paper/2106.11251/) (2021)
- [Self-supervised Product Quantization for Deep Unsupervised Image Retrieval](https://mlsanity.com/paper/2109.02244/) (2021)
- [CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval](https://mlsanity.com/paper/2202.07247/) (2022)
- [Extending CLIP for Category-to-image Retrieval in E-commerce](https://mlsanity.com/paper/2112.11294/) (2021)
- [Cross-Market Product Recommendation](https://mlsanity.com/paper/2109.05929/) (2021)
- [A Transformer-based Embedding Model for Personalized Product Search](https://mlsanity.com/paper/2005.08936/) (2020)
- [Learning Latent Vector Spaces for Product Search](https://mlsanity.com/paper/1608.07253/) (2016)
- [Attentive Long Short-Term Preference Modeling for Personalized Product Search](https://mlsanity.com/paper/1811.10155/) (2018)
- [e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce](https://mlsanity.com/paper/2207.00208/) (2022)
- [PSSL: Self-supervised Learning for Personalized Search with Contrastive Sampling](https://mlsanity.com/paper/2111.12614/) (2021)
- [MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2301.12646/) (2023)

## Abstract (arXiv)

E-commerce click-stream data and product catalogs offer critical user behavior insights and product knowledge. This paper propose a multi-modal transformer termed as PINCER, that leverages the above data sources to transform initial user queries into pseudo-product representations. By tapping into these external data sources, our model can infer users' potential purchase intent from their limited queries and capture query relevant product features. We demonstrate our model's superior performance over state-of-the-art alternatives on e-commerce online retrieval in both controlled and real-world experiments. Our ablation studies confirm that the proposed transformer architecture and integrated learning strategies enable the mining of key data sources to infer purchase intent, extract product features, and enhance the transformation pipeline from queries to more accurate pseudo-product representations.