# Large-Scale Open-Set Classification Protocols for ImageNet

arXiv: https://arxiv.org/abs/2210.06789 · HTML: https://mlsanity.com/paper/2210.06789/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Reducing Network Agnostophobia](https://mlsanity.com/paper/1811.04110/) (2018)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [Learning Placeholders for Open-Set Recognition](https://mlsanity.com/paper/2103.15086/) (2021)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [The Extreme Value Machine](https://mlsanity.com/paper/1506.06112/) (2015)
- [Open-Category Classification by Adversarial Sample Generation](https://mlsanity.com/paper/1705.08722/) (2017)

## Built on by (most cited first, 7 of 7)

- [Learning Transferable Negative Prompts for Out-of-Distribution Detection](https://mlsanity.com/paper/2404.03248/) (2024)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [Open-Set Face Recognition with Maximal Entropy and Objectosphere Loss](https://mlsanity.com/paper/2311.00400/) (2023)
- [A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models](https://mlsanity.com/paper/2501.18463/) (2025)
- [Large-Scale Evaluation of Open-Set Image Classification Techniques](https://mlsanity.com/paper/2406.09112/) (2024)
- [Data-Driven Hierarchical Open Set Recognition](https://mlsanity.com/paper/2411.02635/) (2024)
- [Bias Mitigation in Face Recognition via Demographic-based Supervised Contrastive Learning](https://mlsanity.com/paper/2608.12971/) (2026)

## Abstract (arXiv)

Open-Set Classification (OSC) intends to adapt closed-set classification models to real-world scenarios, where the classifier must correctly label samples of known classes while rejecting previously unseen unknown samples. Only recently, research started to investigate on algorithms that are able to handle these unknown samples correctly. Some of these approaches address OSC by including into the training set negative samples that a classifier learns to reject, expecting that these data increase the robustness of the classifier on unknown classes. Most of these approaches are evaluated on small-scale and low-resolution image datasets like MNIST, SVHN or CIFAR, which makes it difficult to assess their applicability to the real world, and to compare them among each other. We propose three open-set protocols that provide rich datasets of natural images with different levels of similarity between known and unknown classes. The protocols consist of subsets of ImageNet classes selected to provide training and testing data closer to real-world scenarios. Additionally, we propose a new validation metric that can be employed to assess whether the training of deep learning models addresses both the classification of known samples and the rejection of unknown samples. We use the protocols to compare the performance of two baseline open-set algorithms to the standard SoftMax baseline and find that the algorithms work well on negative samples that have been seen during training, and partially on out-of-distribution detection tasks, but drop performance in the presence of samples from previously unseen unknown classes.