# Embedding Arithmetic of Multimodal Queries for Image Retrieval

arXiv: https://arxiv.org/abs/2112.03162 · HTML: https://mlsanity.com/paper/2112.03162/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Automatic Spatially-aware Fashion Concept Discovery](https://mlsanity.com/paper/1708.01311/) (2017)
- [Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language](https://mlsanity.com/paper/1810.11919/) (2018)
- [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019)
- [Compositional Learning of Image-Text Query for Image Retrieval](https://mlsanity.com/paper/2006.11149/) (2020)
- [Finding beans in burgers: Deep semantic-visual embedding with localization](https://mlsanity.com/paper/1804.01720/) (2018)
- [A Benchmark and Baseline for Language-Driven Image Editing](https://mlsanity.com/paper/2010.02330/) (2020)
- [Analogies minus analogy test: measuring regularities in word embeddings](https://mlsanity.com/paper/2010.03446/) (2020)
- [TediGAN: Text-Guided Diverse Face Image Generation and Manipulation](https://mlsanity.com/paper/2012.03308/) (2020)
- [Semantic Layout Manipulation with High-Resolution Sparse Attention](https://mlsanity.com/paper/2012.07288/) (2020)
- [Let's Transfer Transformations of Shared Semantic Representations](https://mlsanity.com/paper/1903.00793/) (2019)

## Built on by (most cited first, 21 of 21)

- [SuS-X: Training-Free Name-Only Transfer of Vision-Language Models](https://mlsanity.com/paper/2211.16198/) (2022)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [Data Roaming and Quality Assessment for Composed Image Retrieval](https://mlsanity.com/paper/2303.09429/) (2023)
- [Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder](https://mlsanity.com/paper/2305.16304/) (2023)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion](https://mlsanity.com/paper/2408.12692/) (2024)
- [Emergent Visual-Semantic Hierarchies in Image-Text Representations](https://mlsanity.com/paper/2407.08521/) (2024)
- [Cross-modal Retrieval for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2401.05736/) (2024)
- [Leveraging Large Language Models for Multimodal Search](https://mlsanity.com/paper/2404.15790/) (2024)
- [BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification](https://mlsanity.com/paper/2203.01937/) (2022)
- [Emergent World Representations in OpenVLA](https://mlsanity.com/paper/2509.24559/) (2025)
- [Concept Lancet: Image Editing with Compositional Representation Transplant](https://mlsanity.com/paper/2504.02828/) (2025)
- [CapS-Adapter: Caption-based MultiModal Adapter in Zero-Shot Classification](https://mlsanity.com/paper/2405.16591/) (2024)
- [They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias](https://mlsanity.com/paper/2406.11331/) (2024)
- [Context-Aware Multimodal Pretraining](https://mlsanity.com/paper/2411.15099/) (2024)
- [Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales](https://mlsanity.com/paper/2605.28036/) (2026)
- [Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning](https://mlsanity.com/paper/2407.15894/) (2024)
- [Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare](https://mlsanity.com/paper/2505.08818/) (2025)
- [Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)](https://mlsanity.com/paper/2508.20322/) (2025)
- [A Surrogate-Assisted Extended Generative Adversarial Network for Parameter Optimization in Free-Form Metasurface Design](https://mlsanity.com/paper/2401.02961/) (2023)
- [Embedding Arithmetic: A Lightweight, Tuning-Free Framework for Post-hoc Bias Mitigation in Text-to-Image Models](https://mlsanity.com/paper/2604.18167/) (2026)

## Abstract (arXiv)

Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing pairwise relationships between scene elements. The goal is to retrieve an image consistent with the (source image, text transformation) query. We use an image/text matching oracle (OSCAR) to assess whether the image transformation is successful. The SIMAT dataset will be publicly available. We use SIMAT to evaluate the geometric properties of multimodal embedding spaces trained with an image/text matching objective, like CLIP. We show that vanilla CLIP embeddings are not very well suited to transform images with delta vectors, but that a simple finetuning on the COCO dataset can bring dramatic improvements. We also study whether it is beneficial to leverage pretrained universal sentence encoders (FastText, LASER and LaBSE).