# Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrieval

arXiv: https://arxiv.org/abs/2501.15379 · HTML: https://mlsanity.com/paper/2501.15379/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline](https://mlsanity.com/paper/1912.02379/) (2019)
- [Conversational Fashion Image Retrieval via Multiturn Natural Language Feedback](https://mlsanity.com/paper/2106.04128/) (2021)
- [Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond](https://mlsanity.com/paper/2402.10805/) (2024)
- [Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach](https://mlsanity.com/paper/2406.03411/) (2024)
- [Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval](https://mlsanity.com/paper/2108.02417/) (2021)
- [Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models](https://mlsanity.com/paper/2404.18746/) (2024)
- [MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval](https://mlsanity.com/paper/2309.01516/) (2023)
- [CFIR: Fast and Effective Long-Text To Image Retrieval for Large Corpora](https://mlsanity.com/paper/2402.15276/) (2024)
- [BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2408.05926/) (2024)
- [3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting](https://mlsanity.com/paper/2404.17273/) (2024)

## Built on by (most cited first, 5 of 5)

- [A Comprehensive Survey on Composed Image Retrieval](https://mlsanity.com/paper/2502.18495/) (2025)
- [Eliminating Hallucination in Diffusion-Augmented Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2601.20391/) (2026)
- [GaiaFlow: Semantic-Guided Diffusion Tuning for Carbon-Frugal Search](https://mlsanity.com/paper/2602.15423/) (2026)
- [ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2603.21886/) (2026)
- [Memory-Augmented Query Intent Understanding for Efficient Chat-based Image Retrieval](https://mlsanity.com/paper/2605.17365/) (2026)

## Abstract (arXiv)

Interactive Text-to-image retrieval (I-TIR) is an important enabler for a wide range of state-of-the-art services in domains such as e-commerce and education. However, current methods rely on finetuned Multimodal Large Language Models (MLLMs), which are costly to train and update, and exhibit poor generalizability. This latter issue is of particular concern, as: 1) finetuning narrows the pretrained distribution of MLLMs, thereby reducing generalizability; and 2) I-TIR introduces increasing query diversity and complexity. As a result, I-TIR solutions are highly likely to encounter queries and images not well represented in any training dataset. To address this, we propose leveraging Diffusion Models (DMs) for text-to-image mapping, to avoid finetuning MLLMs while preserving robust performance on complex queries. Specifically, we introduce Diffusion Augmented Retrieval (DAR), a framework that generates multiple intermediate representations via LLM-based dialogue refinements and DMs, producing a richer depiction of the user's information needs. This augmented representation facilitates more accurate identification of semantically and visually related images. Extensive experiments on four benchmarks show that for simple queries, DAR achieves results on par with finetuned I-TIR models, yet without incurring their tuning overhead. Moreover, as queries become more complex through additional conversational turns, DAR surpasses finetuned I-TIR models by up to 7.61% in Hits@10 after ten turns, illustrating its improved generalization for more intricate queries.