# TALISMAN: Targeted Active Learning for Object Detection with Rare Classes and Slices using Submodular Mutual Information

arXiv: https://arxiv.org/abs/2112.00166 · HTML: https://mlsanity.com/paper/2112.00166/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [Lazier Than Lazy Greedy](https://mlsanity.com/paper/1409.7938/) (2014)
- [Submodular Combinatorial Information Measures with Applications in Machine Learning](https://mlsanity.com/paper/2006.15412/) (2020)
- [SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios](https://mlsanity.com/paper/2107.00717/) (2021)
- [Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Vision](https://mlsanity.com/paper/1901.01151/) (2019)
- [Scalable Active Learning for Object Detection](https://mlsanity.com/paper/2004.04699/) (2020)
- [Active Learning for Deep Object Detection](https://mlsanity.com/paper/1809.09875/) (2018)
- [Localization-Aware Active Learning for Object Detection](https://mlsanity.com/paper/1801.05124/) (2018)
- [Effective Evaluation of Deep Active Learning on Image Classification Tasks](https://mlsanity.com/paper/2106.15324/) (2021)
- [Submodular Functions: from Discrete to Continous Domains](https://mlsanity.com/paper/1511.00394/) (2015)
- [An Adaptive Supervision Framework for Active Learning in Object Detection](https://mlsanity.com/paper/1908.02454/) (2019)
- [Submodularity in Action: From Machine Learning to Signal Processing Applications](https://mlsanity.com/paper/2006.09905/) (2020)
- [A Memoization Framework for Scaling Submodular Optimization to Large Scale Problems](https://mlsanity.com/paper/1902.10176/) (2019)

## Built on by (most cited first, 27 of 27)

- [A Survey on Deep Active Learning: Recent Advances and New Frontiers](https://mlsanity.com/paper/2405.00334/) (2024)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [Coreset Selection for Object Detection](https://mlsanity.com/paper/2404.09161/) (2024)
- [Exploring Diversity-based Active Learning for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/2205.07708/) (2022)
- [CLINICAL: Targeted Active Learning for Imbalanced Medical Image Classification](https://mlsanity.com/paper/2210.01520/) (2022)
- [SMILe: Leveraging Submodular Mutual Information For Robust Few-Shot Object Detection](https://mlsanity.com/paper/2407.02665/) (2024)
- [SCoRe: Submodular Combinatorial Representation Learning](https://mlsanity.com/paper/2310.00165/) (2023)
- [Active Data Discovery: Mining Unknown Data using Submodular Information Measures](https://mlsanity.com/paper/2206.08566/) (2022)
- [Beyond Active Learning: Leveraging the Full Potential of Human Interaction via Auto-Labeling, Human Correction, and Human Verification](https://mlsanity.com/paper/2306.01277/) (2023)
- [Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection](https://mlsanity.com/paper/2510.00303/) (2025)
- [PLATINUM: Semi-Supervised Model Agnostic Meta-Learning using Submodular Mutual Information](https://mlsanity.com/paper/2201.12928/) (2022)
- [MUS-CDB: Mixed Uncertainty Sampling with Class Distribution Balancing for Active Annotation in Aerial Object Detection](https://mlsanity.com/paper/2212.02804/) (2022)
- [Streamlining the Development of Active Learning Methods in Real-World Object Detection](https://mlsanity.com/paper/2508.19906/) (2025)
- [STREAMLINE: Streaming Active Learning for Realistic Multi-Distributional Settings](https://mlsanity.com/paper/2305.10643/) (2023)
- [Two-Step Active Learning for Instance Segmentation with Uncertainty and Diversity Sampling](https://mlsanity.com/paper/2309.16139/) (2023)
- [Theoretical Analysis of Submodular Information Measures for Targeted Data Subset Selection](https://mlsanity.com/paper/2402.13454/) (2024)
- [STONE: A Submodular Optimization Framework for Active 3D Object Detection](https://mlsanity.com/paper/2410.03918/) (2024)
- [InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity](https://mlsanity.com/paper/2508.21003/) (2025)
- [DynaAct: Large Language Model Reasoning with Dynamic Action Spaces](https://mlsanity.com/paper/2511.08043/) (2025)
- [SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition](https://mlsanity.com/paper/2602.05162/) (2026)
- [SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning](https://mlsanity.com/paper/2605.12872/) (2026)
- [BASIL: Balanced Active Semi-supervised Learning for Class Imbalanced Datasets](https://mlsanity.com/paper/2203.05651/) (2022)
- [STENCIL: Submodular Mutual Information Based Weak Supervision for Cold-Start Active Learning](https://mlsanity.com/paper/2402.13468/) (2024)
- [Complement Submodular Information Measures for Balanced and Robust Data Selection](https://mlsanity.com/paper/2605.24779/) (2026)
- [Understanding Submodular Information Measure Based Objectives for Representation Learning: A Variance and Separation Perspective](https://mlsanity.com/paper/2607.27660/) (2026)
- [Distance to Class Prototypes: Active Learning for Object Detection](https://mlsanity.com/paper/2609.20248/) (2026)

## Abstract (arXiv)

Deep neural networks based object detectors have shown great success in a variety of domains like autonomous vehicles, biomedical imaging, etc. It is known that their success depends on a large amount of data from the domain of interest. While deep models often perform well in terms of overall accuracy, they often struggle in performance on rare yet critical data slices. For example, data slices like "motorcycle at night" or "bicycle at night" are often rare but very critical slices for self-driving applications and false negatives on such rare slices could result in ill-fated failures and accidents. Active learning (AL) is a well-known paradigm to incrementally and adaptively build training datasets with a human in the loop. However, current AL based acquisition functions are not well-equipped to tackle real-world datasets with rare slices, since they are based on uncertainty scores or global descriptors of the image. We propose TALISMAN, a novel framework for Targeted Active Learning or object detectIon with rare slices using Submodular MutuAl iNformation. Our method uses the submodular mutual information functions instantiated using features of the region of interest (RoI) to efficiently target and acquire data points with rare slices. We evaluate our framework on the standard PASCAL VOC07+12 and BDD100K, a real-world self-driving dataset. We observe that TALISMAN outperforms other methods by in terms of average precision on rare slices, and in terms of mAP.