# DashCLIP: Leveraging multimodal models for generating semantic embeddings for DoorDash

arXiv: https://arxiv.org/abs/2504.07110 · HTML: https://mlsanity.com/paper/2504.07110/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://mlsanity.com/paper/1806.03514/) (2018)
- [ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest](https://mlsanity.com/paper/2205.11728/) (2022)
- [Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta](https://mlsanity.com/paper/2311.09544/) (2023)
- [Adversarial Multimodal Representation Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2003.07162/) (2020)
- [General-Purpose User Embeddings based on Mobile App Usage](https://mlsanity.com/paper/2005.13303/) (2020)
- [Visual Encoding and Debiasing for CTR Prediction](https://mlsanity.com/paper/2205.04168/) (2022)

## Built on by (most cited first, 2 of 2)

- [To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay](https://mlsanity.com/paper/2505.04209/) (2025)
- [LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBay](https://mlsanity.com/paper/2508.03628/) (2025)

## Abstract (arXiv)

Despite the success of vision-language models in various generative tasks, obtaining high-quality semantic representations for products and user intents is still challenging due to the inability of off-the-shelf models to capture nuanced relationships between the entities. In this paper, we introduce a joint training framework for product and user queries by aligning uni-modal and multi-modal encoders through contrastive learning on image-text data. Our novel approach trains a query encoder with an LLM-curated relevance dataset, eliminating the reliance on engagement history. These embeddings demonstrate strong generalization capabilities and improve performance across applications, including product categorization and relevance prediction. For personalized ads recommendation, a significant uplift in the click-through rate and conversion rate after the deployment further confirms the impact on key business metrics. We believe that the flexibility of our framework makes it a promising solution toward enriching the user experience across the e-commerce landscape.