# Dialog-based Interactive Image Retrieval

arXiv: https://arxiv.org/abs/1805.00145 · HTML: https://mlsanity.com/paper/1805.00145/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2018.

## Built on by (most cited first, 30 of 84)

- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Zero-Shot Composed Image Retrieval with Textual Inversion](https://mlsanity.com/paper/2303.15247/) (2023)
- [CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion](https://mlsanity.com/paper/2303.11916/) (2023)
- [ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity](https://mlsanity.com/paper/2203.08101/) (2022)
- [Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features](https://mlsanity.com/paper/2308.11485/) (2023)
- [CoVR-2: Automatic Data Construction for Composed Video Retrieval](https://mlsanity.com/paper/2308.14746/) (2023)
- [Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.09738/) (2024)
- [Data Roaming and Quality Assessment for Composed Image Retrieval](https://mlsanity.com/paper/2303.09429/) (2023)
- [Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization](https://mlsanity.com/paper/2211.07394/) (2022)
- [Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2405.00571/) (2024)
- [FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval](https://mlsanity.com/paper/2503.21309/) (2025)
- [OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrieval](https://mlsanity.com/paper/2507.05631/) (2025)
- [Chat-Driven Text Generation and Interaction for Person Retrieval](https://mlsanity.com/paper/2509.12662/) (2025)
- [FashionViL: Fashion-Focused Vision-and-Language Representation Learning](https://mlsanity.com/paper/2207.08150/) (2022)
- [Conversational Fashion Image Retrieval via Multiturn Natural Language Feedback](https://mlsanity.com/paper/2106.04128/) (2021)
- [Probabilistic Compositional Embeddings for Multimodal Image Retrieval](https://mlsanity.com/paper/2204.05845/) (2022)
- [The 5th AI City Challenge](https://mlsanity.com/paper/2104.12233/) (2021)
- [RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network](https://mlsanity.com/paper/2104.03015/) (2021)
- [CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions](https://mlsanity.com/paper/2101.04741/) (2021)
- [Progressive Learning for Image Retrieval with Hybrid-Modality Queries](https://mlsanity.com/paper/2204.11212/) (2022)
- [Chatting Makes Perfect: Chat-based Image Retrieval](https://mlsanity.com/paper/2305.20062/) (2023)
- [Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach](https://mlsanity.com/paper/2406.03411/) (2024)
- [iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2405.02951/) (2024)
- [HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval](https://mlsanity.com/paper/2604.18037/) (2026)
- [The 6th AI City Challenge](https://mlsanity.com/paper/2204.10380/) (2022)
- [CaLa: Complementary Association Learning for Augmenting Composed Image Retrieval](https://mlsanity.com/paper/2405.19149/) (2024)
- [FreestyleRet: Retrieving Images from Style-Diversified Queries](https://mlsanity.com/paper/2312.02428/) (2023)
- [Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models](https://mlsanity.com/paper/2404.18746/) (2024)
- [A Comprehensive Survey on Composed Image Retrieval](https://mlsanity.com/paper/2502.18495/) (2025)
- [Geo-Localization via Ground-to-Satellite Cross-View Image Retrieval](https://mlsanity.com/paper/2205.10878/) (2022)

## Abstract (arXiv)

Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.