# Object-Level Targeted Selection via Deep Template Matching

arXiv: https://arxiv.org/abs/2207.01778 · HTML: https://mlsanity.com/paper/2207.01778/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 50 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [Learning to Compare Image Patches via Convolutional Neural Networks](https://mlsanity.com/paper/1504.03641/) (2015)
- [MultiGrain: a unified image embedding for classes and instances](https://mlsanity.com/paper/1902.05509/) (2019)
- [Learning and aggregating deep local descriptors for instance-level recognition](https://mlsanity.com/paper/2007.13172/) (2020)
- [Visual Instance Retrieval with Deep Convolutional Networks](https://mlsanity.com/paper/1412.6574/) (2014)
- [SOLAR: Second-Order Loss and Attention for Image Retrieval](https://mlsanity.com/paper/2001.08972/) (2020)
- [Cross-dimensional Weighting for Aggregated Deep Convolutional Features](https://mlsanity.com/paper/1512.04065/) (2015)
- [Detect-to-Retrieve: Efficient Regional Aggregation for Image Search](https://mlsanity.com/paper/1812.01584/) (2018)
- [Multi-scale Orderless Pooling of Deep Convolutional Activation Features](https://mlsanity.com/paper/1403.1840/) (2014)
- [Multi-Cue Zero-Shot Learning with Strong Supervision](https://mlsanity.com/paper/1603.08754/) (2016)
- [Graph-RISE: Graph-Regularized Image Semantic Embedding](https://mlsanity.com/paper/1902.10814/) (2019)
- [Hybrid-Attention based Decoupled Metric Learning for Zero-Shot Image Retrieval](https://mlsanity.com/paper/1907.11832/) (2019)
- [Best-Buddies Similarity - Robust Template Matching using Mutual Nearest Neighbors](https://mlsanity.com/paper/1609.01571/) (2016)
- [Enhancing Remote Sensing Image Retrieval with Triplet Deep Metric Learning Network](https://mlsanity.com/paper/1902.05818/) (2019)
- [QATM: Quality-Aware Template Matching For Deep Learning](https://mlsanity.com/paper/1903.07254/) (2019)
- [Template Matching with Deformable Diversity Similarity](https://mlsanity.com/paper/1612.02190/) (2016)

## Abstract (arXiv)

Retrieving images with objects that are semantically similar to objects of interest (OOI) in a query image has many practical use cases. A few examples include fixing failures like false negatives/positives of a learned model or mitigating class imbalance in a dataset. The targeted selection task requires finding the relevant data from a large-scale pool of unlabeled data. Manual mining at this scale is infeasible. Further, the OOI are often small and occupy less than 1% of image area, are occluded, and co-exist with many semantically different objects in cluttered scenes. Existing semantic image retrieval methods often focus on mining for larger sized geographical landmarks, and/or require extra labeled data, such as images/image-pairs with similar objects, for mining images with generic objects. We propose a fast and robust template matching algorithm in the DNN feature space, that retrieves semantically similar images at the object-level from a large unlabeled pool of data. We project the region(s) around the OOI in the query image to the DNN feature space for use as the template. This enables our method to focus on the semantics of the OOI without requiring extra labeled data. In the context of autonomous driving, we evaluate our system for targeted selection by using failure cases of object detectors as OOI. We demonstrate its efficacy on a large unlabeled dataset with 2.2M images and show high recall in mining for images with small-sized OOI. We compare our method against a well-known semantic image retrieval method, which also does not require extra labeled data. Lastly, we show that our method is flexible and retrieves images with one or more semantically different co-occurring OOI seamlessly.