# Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop

arXiv: https://arxiv.org/abs/1512.05227 · HTML: https://mlsanity.com/paper/1512.05227/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2015.

## Built on by (most cited first, 25 of 25)

- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Webly Supervised Fine-Grained Recognition: Benchmark Datasets and An Approach](https://mlsanity.com/paper/2108.02399/) (2021)
- [Learning a Structured Latent Space for Unsupervised Point Cloud Completion](https://mlsanity.com/paper/2203.15580/) (2022)
- [COARSE3D: Class-Prototypes for Contrastive Learning in Weakly-Supervised 3D Point Cloud Segmentation](https://mlsanity.com/paper/2210.01784/) (2022)
- [Data Augmentation for Seizure Prediction with Generative Diffusion Model](https://mlsanity.com/paper/2306.08256/) (2023)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [The Curious Layperson: Fine-Grained Image Recognition without Expert Labels](https://mlsanity.com/paper/2111.03651/) (2021)
- [Trends in Vehicle Re-identification Past, Present, and Future: A Comprehensive Review](https://mlsanity.com/paper/2102.09744/) (2021)
- [Making a Bird AI Expert Work for You and Me](https://mlsanity.com/paper/2112.02747/) (2021)
- [Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images](https://mlsanity.com/paper/2401.14322/) (2024)
- [Self-Supervised Endoscopic Image Key-Points Matching](https://mlsanity.com/paper/2208.11424/) (2022)
- [Multi-Granularity Archaeological Dating of Chinese Bronze Dings Based on a Knowledge-Guided Relation Graph](https://mlsanity.com/paper/2303.15266/) (2023)
- [Interactive Acquisition of Fine-grained Visual Concepts by Exploiting Semantics of Generic Characterizations in Discourse](https://mlsanity.com/paper/2305.03461/) (2023)
- [Construct Informative Triplet with Two-stage Hard-sample Generation](https://mlsanity.com/paper/2112.02259/) (2021)
- [Urban feature analysis from aerial remote sensing imagery using self-supervised and semi-supervised computer vision](https://mlsanity.com/paper/2208.08047/) (2022)
- [Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification](https://mlsanity.com/paper/2201.11388/) (2022)
- [Relational Proxies: Emergent Relationships as Fine-Grained Discriminators](https://mlsanity.com/paper/2210.02149/) (2022)
- [Exploiting Web Images for Fine-Grained Visual Recognition by Eliminating Noisy Samples and Utilizing Hard Ones](https://mlsanity.com/paper/2101.09412/) (2021)
- [Improving Image co-segmentation via Deep Metric Learning](https://mlsanity.com/paper/2103.10670/) (2021)
- [Learning a Discriminant Latent Space with Neural Discriminant Analysis](https://mlsanity.com/paper/2107.06209/) (2021)
- [Transitivity Recovering Decompositions: Interpretable and Robust Fine-Grained Relationships](https://mlsanity.com/paper/2310.15999/) (2023)
- [DAAL: Density-Aware Adaptive Line Margin Loss for Multi-Modal Deep Metric Learning](https://mlsanity.com/paper/2410.05438/) (2024)
- [Scalable Deep Metric Learning on Attributed Graphs](https://mlsanity.com/paper/2411.13014/) (2024)
- [LCA: Local Classifier Alignment for Continual Learning](https://mlsanity.com/paper/2603.09888/) (2026)
- [MD-ProTector: Positioning Multiple Data-Driven Prototypes for LLM-Generated Text Detection](https://mlsanity.com/paper/2608.10459/) (2026)

## Abstract (arXiv)

Existing fine-grained visual categorization methods often suffer from three challenges: lack of training data, large number of fine-grained categories, and high intraclass vs. low inter-class variance. In this work we propose a generic iterative framework for fine-grained categorization and dataset bootstrapping that handles these three challenges. Using deep metric learning with humans in the loop, we learn a low dimensional feature embedding with anchor points on manifolds for each category. These anchor points capture intra-class variances and remain discriminative between classes. In each round, images with high confidence scores from our model are sent to humans for labeling. By comparing with exemplar images, labelers mark each candidate image as either a "true positive" or a "false positive". True positives are added into our current dataset and false positives are regarded as "hard negatives" for our metric learning model. Then the model is retrained with an expanded dataset and hard negatives for the next round. To demonstrate the effectiveness of the proposed framework, we bootstrap a fine-grained flower dataset with 620 categories from Instagram images. The proposed deep metric learning scheme is evaluated on both our dataset and the CUB-200-2001 Birds dataset. Experimental evaluations show significant performance gain using dataset bootstrapping and demonstrate state-of-the-art results achieved by the proposed deep metric learning methods.