Query-based Hard-Image Retrieval for Object Detection at Test Time
Not on the map. Why?
Abstract (arXiv)
There is a longstanding interest in capturing the error behaviour of object detectors by finding images where their performance is likely to be unsatisfactory. In real-world applications such as autonomous driving, it is also crucial to characterise potential failures beyond simple requirements of detection performance. For example, a missed detection of a pedestrian close to an ego vehicle will generally require closer inspection than a missed detection of a car in the distance. The problem of predicting such potential failures at test time has largely been overlooked in the literature and conventional approaches based on detection uncertainty fall short in that they are agnostic to such fine-grained characterisation of errors. In this work, we propose to reformulate the problem of finding "hard" images as a query-based hard image retrieval task, where queries are specific definitions of "hardness", and offer a simple and intuitive method that can solve this task for a large family of queries. Our method is entirely post-hoc, does not require ground-truth annotations, is independent of the choice of a detector, and relies on an efficient Monte Carlo estimation that uses a simple stochastic model in place of the ground-truth. We show experimentally that it can be applied successfully to a wide variety of queries for which it can reliably identify hard images for a given detector without any labelled data. We provide results on ranking and classification tasks using the widely used RetinaNet, Faster-RCNN, Mask-RCNN, and Cascade Mask-RCNN object detectors. The code for this project is available at https://github.com/fiveai/hardest.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 23 Sep 2022.
Builds on
21 of 33 references.
References · 21
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2017Mask R-CNNcited by 7,277
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
- 2017Deep Bayesian Active Learning with Image Datacited by 819
- 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
- 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
- 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
- 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
- 2020Calibrating Deep Neural Networks using Focal Losscited by 357
9 more
- 2019Libra R-CNN: Towards Balanced Learning for Object Detectioncited by 255
- 2021Active Learning for Deep Object Detection via Probabilistic Modelingcited by 79
- 2019Active Learning for Deep Detection Neural Networkscited by 62
- 2020Scalable Active Learning for Object Detectioncited by 58
- 2019Deep Active Learning for Efficient Training of a LiDAR 3D Object Detectorcited by 45
- 2021Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectorscited by 37
- 2021Consistency-based Active Learning for Object Detectioncited by 32
- 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
- 2016Hard Negative Mining for Metric Learning Based Zero-Shot Classificationcited by 18
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.
2 papers.