# Structured Visual Search via Composition-aware Learning

arXiv: https://arxiv.org/abs/2010.14438 · HTML: https://mlsanity.com/paper/2010.14438/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Uni-Retrieval: A Multi-Style Retrieval Framework for STEM's Education](https://mlsanity.com/paper/2502.05863/) (2025)
- [Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval](https://mlsanity.com/paper/2102.04980/) (2021)
- [Human-Object Interaction Detection via Weak Supervision](https://mlsanity.com/paper/2112.00492/) (2021)
- [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024)
- [LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval](https://mlsanity.com/paper/2606.05489/) (2026)

## Abstract (arXiv)

This paper studies visual search using structured queries. The structure is in the form of a 2D composition that encodes the position and the category of the objects. The transformation of the position and the category of the objects leads to a continuous-valued relationship between visual compositions, which carries highly beneficial information, although not leveraged by previous techniques. To that end, in this work, our goal is to leverage these continuous relationships by using the notion of symmetry in equivariance. Our model output is trained to change symmetrically with respect to the input transformations, leading to a sensitive feature space. Doing so leads to a highly efficient search technique, as our approach learns from fewer data using a smaller feature space. Experiments on two large-scale benchmarks of MS-COCO and HICO-DET demonstrates that our approach leads to a considerable gain in the performance against competing techniques.