# I Want This Product but Different : Multimodal Retrieval with Synthetic Query Expansion

arXiv: https://arxiv.org/abs/2102.08871 · HTML: https://mlsanity.com/paper/2102.08871/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language](https://mlsanity.com/paper/1810.11919/) (2018)
- [Dialog-based Interactive Image Retrieval](https://mlsanity.com/paper/1805.00145/) (2018)
- [Learning Fashion Compatibility with Bidirectional LSTMs](https://mlsanity.com/paper/1707.05691/) (2017)
- [Cross-domain Image Retrieval with a Dual Attribute-aware Ranking Network](https://mlsanity.com/paper/1505.07922/) (2015)
- [Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback](https://mlsanity.com/paper/1905.12794/) (2019)
- [Multimodal Residual Learning for Visual QA](https://mlsanity.com/paper/1606.01455/) (2016)
- [Sequential Attention GAN for Interactive Image Editing](https://mlsanity.com/paper/1812.08352/) (2018)
- [Visual Search at Alibaba](https://mlsanity.com/paper/2102.04674/) (2021)
- [Creating Capsule Wardrobes from Fashion Images](https://mlsanity.com/paper/1712.02662/) (2017)
- [Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images](https://mlsanity.com/paper/1707.03376/) (2017)
- [End-to-end optimization of goal-driven and visually grounded dialogue systems](https://mlsanity.com/paper/1703.05423/) (2017)
- [DeepStyle: Multimodal Search Engine for Fashion and Interior Design](https://mlsanity.com/paper/1801.03002/) (2018)
- [Web-Scale Responsive Visual Search at Bing](https://mlsanity.com/paper/1802.04914/) (2018)

## Built on by (most cited first, 4 of 4)

- [A Comprehensive Survey on Composed Image Retrieval](https://mlsanity.com/paper/2502.18495/) (2025)
- [Dynamic Weighted Combiner for Mixed-Modal Image Retrieval](https://mlsanity.com/paper/2312.06179/) (2023)
- [Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search](https://mlsanity.com/paper/2402.07742/) (2024)
- [Language Guided Local Infiltration for Interactive Image Retrieval](https://mlsanity.com/paper/2304.07747/) (2023)

## Abstract (arXiv)

This paper addresses the problem of media retrieval using a multimodal query (a query which combines visual input with additional semantic information in natural language feedback). We propose a SynthTriplet GAN framework which resolves this task by expanding the multimodal query with a synthetically generated image that captures semantic information from both image and text input. We introduce a novel triplet mining method that uses a synthetic image as an anchor to directly optimize for embedding distances of generated and target images. We demonstrate that apart from the added value of retrieval illustration with synthetic image with the focus on customization and user feedback, the proposed method greatly surpasses other multimodal generation methods and achieves state of the art results in the multimodal retrieval task. We also show that in contrast to other retrieval methods, our method provides explainable embeddings.