# Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query

arXiv: https://arxiv.org/abs/2103.01654 · HTML: https://mlsanity.com/paper/2103.01654/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Dialog-based Interactive Image Retrieval](https://mlsanity.com/paper/1805.00145/) (2018)
- [Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.06585/) (2017)
- [Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching](https://mlsanity.com/paper/1909.11416/) (2019)
- [Position Focused Attention Network for Image-Text Matching](https://mlsanity.com/paper/1907.09748/) (2019)
- [Saliency-Guided Attention Network for Image-Sentence Matching](https://mlsanity.com/paper/1904.09471/) (2019)
- [Drill-down: Interactive Retrieval of Complex Scenes using Natural Language Queries](https://mlsanity.com/paper/1911.03826/) (2019)
- [Interactive Video Retrieval with Dialog](https://mlsanity.com/paper/1905.02442/) (2019)
- [Dialog Policy Learning for Joint Clarification and Active Learning Queries](https://mlsanity.com/paper/2006.05456/) (2020)
- [Image Retrieval with Mixed Initiative and Multimodal Feedback](https://mlsanity.com/paper/1805.03134/) (2018)
- [From A Glance to "Gotcha": Interactive Facial Image Retrieval with Progressive Relevance Feedback](https://mlsanity.com/paper/2007.15683/) (2020)

## Built on by (most cited first, 9 of 9)

- [Learning to Retrieve Videos by Asking Questions](https://mlsanity.com/paper/2205.05739/) (2022)
- [Simple Baselines for Interactive Video Retrieval with Questions and Answers](https://mlsanity.com/paper/2308.10402/) (2023)
- [LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification](https://mlsanity.com/paper/2504.10174/) (2025)
- [Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization](https://mlsanity.com/paper/2507.15504/) (2025)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [Multimodal LLM-based Query Paraphrasing for Video Search](https://mlsanity.com/paper/2407.12341/) (2024)
- [DialogueVPR: Towards Conversational Visual Place Recognition](https://mlsanity.com/paper/2607.14115/) (2026)
- [Click-Feedback Retrieval](https://mlsanity.com/paper/2305.00052/) (2023)
- [ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search](https://mlsanity.com/paper/2604.12762/) (2026)

## Abstract (arXiv)

Text-based image retrieval has seen considerable progress in recent years. However, the performance of existing methods suffers in real life since the user is likely to provide an incomplete description of an image, which often leads to results filled with false positives that fit the incomplete description. In this work, we introduce the partial-query problem and extensively analyze its influence on text-based image retrieval. Previous interactive methods tackle the problem by passively receiving users' feedback to supplement the incomplete query iteratively, which is time-consuming and requires heavy user effort. Instead, we propose a novel retrieval framework that conducts the interactive process in an Ask-and-Confirm fashion, where AI actively searches for discriminative details missing in the current query, and users only need to confirm AI's proposal. Specifically, we propose an object-based interaction to make the interactive retrieval more user-friendly and present a reinforcement-learning-based policy to search for discriminative objects. Furthermore, since fully-supervised training is often infeasible due to the difficulty of obtaining human-machine dialog data, we present a weakly-supervised training strategy that needs no human-annotated dialogs other than a text-image dataset. Experiments show that our framework significantly improves the performance of text-based image retrieval. Code is avaiable at https://github.com/CuthbertCai/Ask-Confirm.