# Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach

arXiv: https://arxiv.org/abs/2406.03411 · HTML: https://mlsanity.com/paper/2406.03411/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [Vision-by-Language for Training-Free Compositional Image Retrieval](https://mlsanity.com/paper/2310.09291/) (2023)
- [Dialog-based Interactive Image Retrieval](https://mlsanity.com/paper/1805.00145/) (2018)

## Built on by (most cited first, 25 of 25)

- [Chat-Driven Text Generation and Interaction for Person Retrieval](https://mlsanity.com/paper/2509.12662/) (2025)
- [Uni-Retrieval: A Multi-Style Retrieval Framework for STEM's Education](https://mlsanity.com/paper/2502.05863/) (2025)
- [ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning](https://mlsanity.com/paper/2503.10166/) (2025)
- [MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline](https://mlsanity.com/paper/2407.12508/) (2024)
- [LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification](https://mlsanity.com/paper/2504.10174/) (2025)
- [Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2501.15379/) (2025)
- [Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval](https://mlsanity.com/paper/2505.15877/) (2025)
- [Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization](https://mlsanity.com/paper/2507.15504/) (2025)
- [Chain-of-Thought Re-ranking for Image Retrieval Tasks](https://mlsanity.com/paper/2509.14746/) (2025)
- [VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval](https://mlsanity.com/paper/2505.20291/) (2025)
- [A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback](https://mlsanity.com/paper/2511.17255/) (2025)
- [From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEM](https://mlsanity.com/paper/2507.03868/) (2025)
- [SA-Person: Text-Based Person Retrieval with Scene-aware Re-ranking](https://mlsanity.com/paper/2505.24466/) (2025)
- [GenIR: Generative Visual Feedback for Mental Image Retrieval](https://mlsanity.com/paper/2506.06220/) (2025)
- [GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrieval](https://mlsanity.com/paper/2509.23370/) (2025)
- [Sherlock Your Queries: Learning to Ask the Right Questions for Dialogue-Based Retrieval](https://mlsanity.com/paper/2510.18659/) (2025)
- [VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension](https://mlsanity.com/paper/2601.12781/) (2026)
- [DialogueVPR: Towards Conversational Visual Place Recognition](https://mlsanity.com/paper/2607.14115/) (2026)
- [Eliminating Hallucination in Diffusion-Augmented Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2601.20391/) (2026)
- [ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2603.21886/) (2026)
- [ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search](https://mlsanity.com/paper/2604.12762/) (2026)
- [Memory-Augmented Query Intent Understanding for Efficient Chat-based Image Retrieval](https://mlsanity.com/paper/2605.17365/) (2026)
- [CIRCLED: A Multi-turn CIR Dataset with Consistent Dialogues across Domains](https://mlsanity.com/paper/2605.26734/) (2026)
- [Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search](https://mlsanity.com/paper/2608.23503/) (2026)
- [What Should We Ask Next? Retrieval-Aware Question Learning for Interactive ReID](https://mlsanity.com/paper/2609.21924/) (2026)

## Abstract (arXiv)

In this paper, we primarily address the issue of dialogue-form context query within the interactive text-to-image retrieval task. Our methodology, PlugIR, actively utilizes the general instruction-following capability of LLMs in two ways. First, by reformulating the dialogue-form context, we eliminate the necessity of fine-tuning a retrieval model on existing visual dialogue data, thereby enabling the use of any arbitrary black-box model. Second, we construct the LLM questioner to generate non-redundant questions about the attributes of the target image, based on the information of retrieval candidate images in the current context. This approach mitigates the issues of noisiness and redundancy in the generated questions. Beyond our methodology, we propose a novel evaluation metric, Best log Rank Integral (BRI), for a comprehensive assessment of the interactive retrieval system. PlugIR demonstrates superior performance compared to both zero-shot and fine-tuned baselines in various benchmarks. Additionally, the two methodologies comprising PlugIR can be flexibly applied together or separately in various situations. Our codes are available at https://github.com/Saehyung-Lee/PlugIR.