# Drill-down: Interactive Retrieval of Complex Scenes using Natural Language Queries

arXiv: https://arxiv.org/abs/1911.03826 · HTML: https://mlsanity.com/paper/1911.03826/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 10 of 10)

- [MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction](https://mlsanity.com/paper/2509.18095/) (2025)
- [Simple Baselines for Interactive Video Retrieval with Questions and Answers](https://mlsanity.com/paper/2308.10402/) (2023)
- [Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query](https://mlsanity.com/paper/2103.01654/) (2021)
- [Estimating and Maximizing Mutual Information for Knowledge Distillation](https://mlsanity.com/paper/2110.15946/) (2021)
- [Interactive Video Corpus Moment Retrieval using Reinforcement Learning](https://mlsanity.com/paper/2302.09522/) (2023)
- [Hierarchical Matching and Reasoning for Multi-Query Image Retrieval](https://mlsanity.com/paper/2306.14460/) (2023)
- [Composed Multi-modal Retrieval: A Survey of Approaches and Applications](https://mlsanity.com/paper/2503.01334/) (2025)
- [Tell Your Story: Task-Oriented Dialogs for Interactive Content Creation](https://mlsanity.com/paper/2211.03940/) (2022)
- [Reinforced Approximate Exploratory Data Analysis](https://mlsanity.com/paper/2212.06225/) (2022)
- [Navigating Connected Memories with a Task-oriented Dialog System](https://mlsanity.com/paper/2211.08462/) (2022)

## Abstract (arXiv)

This paper explores the task of interactive image retrieval using natural language queries, where a user progressively provides input queries to refine a set of retrieval results. Moreover, our work explores this problem in the context of complex image scenes containing multiple objects. We propose Drill-down, an effective framework for encoding multiple queries with an efficient compact state representation that significantly extends current methods for single-round image retrieval. We show that using multiple rounds of natural language queries as input can be surprisingly effective to find arbitrarily specific images of complex scenes. Furthermore, we find that existing image datasets with textual captions can provide a surprisingly effective form of weak supervision for this task. We compare our method with existing sequential encoding and embedding networks, demonstrating superior performance on two proposed benchmarks: automatic image retrieval on a simulated scenario that uses region captions as queries, and interactive image retrieval using real queries from human evaluators.