# Smart Multi-Modal Search: Contextual Sparse and Dense Embedding Integration in Adobe Express

arXiv: https://arxiv.org/abs/2408.14698 · HTML: https://mlsanity.com/paper/2408.14698/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification](https://mlsanity.com/paper/2112.03562/) (2021)
- [Relevance Filtering for Embedding-based Retrieval](https://mlsanity.com/paper/2408.04887/) (2024)
- [Length is a Curse and a Blessing for Document-level Semantics](https://mlsanity.com/paper/2310.16193/) (2023)
- [Combining Embedding-Based and Semantic-Based Models for Post-hoc Explanations in Recommender Systems](https://mlsanity.com/paper/2401.04474/) (2024)
- [Contextual Multilingual Spellchecker for User Queries](https://mlsanity.com/paper/2305.01082/) (2023)
- [Contextual Font Recommendations based on User Intent](https://mlsanity.com/paper/2306.08188/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Matter of Time: Revealing the Structure of Time in Vision-Language Models](https://mlsanity.com/paper/2510.19559/) (2025)

## Abstract (arXiv)

As user content and queries become increasingly multi-modal, the need for effective multi-modal search systems has grown. Traditional search systems often rely on textual and metadata annotations for indexed images, while multi-modal embeddings like CLIP enable direct search using text and image embeddings. However, embedding-based approaches face challenges in integrating contextual features such as user locale and recency. Building a scalable multi-modal search system requires fine-tuning several components. This paper presents a multi-modal search architecture and a series of AB tests that optimize embeddings and multi-modal technologies in Adobe Express template search. We address considerations such as embedding model selection, the roles of embeddings in matching and ranking, and the balance between dense and sparse embeddings. Our iterative approach demonstrates how utilizing sparse, dense, and contextual features enhances short and long query search, significantly reduces null rates (over 70%), and increases click-through rates (CTR). Our findings provide insights into developing robust multi-modal search systems, thereby enhancing relevance for complex queries.