# WhittleSearch: Interactive Image Search with Relative Attribute Feedback

arXiv: https://arxiv.org/abs/1505.04141 · HTML: https://mlsanity.com/paper/1505.04141/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2015.

## Built on by (most cited first, 10 of 10)

- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features](https://mlsanity.com/paper/2308.11485/) (2023)
- [LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification](https://mlsanity.com/paper/2504.10174/) (2025)
- [From Culture to Clothing: Discovering the World Events Behind A Century of Fashion Images](https://mlsanity.com/paper/2102.01690/) (2021)
- [Image Search with Text Feedback by Additive Attention Compositional Learning](https://mlsanity.com/paper/2203.03809/) (2022)
- [TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP](https://mlsanity.com/paper/2505.18434/) (2025)
- [Smart Fashion: A Review of AI Applications in the Fashion & Apparel Industry](https://mlsanity.com/paper/2111.00905/) (2021)
- [Give me a hint! Navigating Image Databases using Human-in-the-loop Feedback](https://mlsanity.com/paper/1809.08714/) (2018)
- [DialogueVPR: Towards Conversational Visual Place Recognition](https://mlsanity.com/paper/2607.14115/) (2026)
- [What Should We Ask Next? Retrieval-Aware Question Learning for Interactive ReID](https://mlsanity.com/paper/2609.21924/) (2026)

## Abstract (arXiv)

We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image sought. For example, perusing image results for a query "black shoes", the user might state, "Show me shoe images like these, but sportier." Offline, our approach first learns a set of ranking functions, each of which predicts the relative strength of a nameable attribute in an image (e.g., sportiness). At query time, the system presents the user with a set of exemplar images, and the user relates them to his/her target image with comparative statements. Using a series of such constraints in the multi-dimensional attribute space, our method iteratively updates its relevance function and re-ranks the database of images. To determine which exemplar images receive feedback from the user, we present two variants of the approach: one where the feedback is user-initiated and another where the feedback is actively system-initiated. In either case, our approach allows a user to efficiently "whittle away" irrelevant portions of the visual feature space, using semantic language to precisely communicate her preferences to the system. We demonstrate our technique for refining image search for people, products, and scenes, and we show that it outperforms traditional binary relevance feedback in terms of search speed and accuracy. In addition, the ordinal nature of relative attributes helps make our active approach efficient -- both computationally for the machine when selecting the reference images, and for the user by requiring less user interaction than conventional passive and active methods.