# Measuring Human Perception to Improve Open Set Recognition

arXiv: https://arxiv.org/abs/2209.03519 · HTML: https://mlsanity.com/paper/2209.03519/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Reducing Network Agnostophobia](https://mlsanity.com/paper/1811.04110/) (2018)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [Classification-Reconstruction Learning for Open-Set Recognition](https://mlsanity.com/paper/1812.04246/) (2018)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [OpenGAN: Open-Set Recognition via Open Data Generation](https://mlsanity.com/paper/2104.02939/) (2021)
- [Learning Placeholders for Open-Set Recognition](https://mlsanity.com/paper/2103.15086/) (2021)
- [The Extreme Value Machine](https://mlsanity.com/paper/1506.06112/) (2015)
- [AGIL: Learning Attention from Human for Visuomotor Tasks](https://mlsanity.com/paper/1806.03960/) (2018)
- [The Familiarity Hypothesis: Explaining the Behavior of Deep Open Set Methods](https://mlsanity.com/paper/2203.02486/) (2022)
- [Visual Psychophysics for Making Face Recognition Algorithms More Explainable](https://mlsanity.com/paper/1803.07140/) (2018)
- [PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition](https://mlsanity.com/paper/1611.06448/) (2016)
- [Deep CNN-based Multi-task Learning for Open-Set Recognition](https://mlsanity.com/paper/1903.03161/) (2019)
- [Measuring Human Perception to Improve Handwritten Document Transcription](https://mlsanity.com/paper/1904.03734/) (2019)
- [Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluations](https://mlsanity.com/paper/2005.06583/) (2020)
- [How Deep is the Feature Analysis underlying Rapid Visual Categorization?](https://mlsanity.com/paper/1606.01167/) (2016)
- [A Unifying Framework for Formal Theories of Novelty:Framework, Examples and Discussion](https://mlsanity.com/paper/2012.04226/) (2020)
- [Predicting First Impressions with Deep Learning](https://mlsanity.com/paper/1610.08119/) (2016)

## Built on by (most cited first, 7 of 7)

- [Teaching AI to Teach: Leveraging Limited Human Salience Data Into Unlimited Saliency-Based Training](https://mlsanity.com/paper/2306.05527/) (2023)
- [A Survey on Open-Set Image Recognition](https://mlsanity.com/paper/2312.15571/) (2023)
- [Using Human Perception to Regularize Transfer Learning](https://mlsanity.com/paper/2211.07885/) (2022)
- [MENTOR: Human Perception-Guided Pretraining for Increased Generalization](https://mlsanity.com/paper/2310.19545/) (2023)
- [Taking Training Seriously: Human Guidance and Management-Based Regulation of Artificial Intelligence](https://mlsanity.com/paper/2402.08466/) (2024)
- [VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception](https://mlsanity.com/paper/2508.09061/) (2025)
- [Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescue](https://mlsanity.com/paper/2412.05553/) (2024)

## Abstract (arXiv)

The human ability to recognize when an object belongs or does not belong to a particular vision task outperforms all open set recognition algorithms. Human perception as measured by the methods and procedures of visual psychophysics from psychology provides an additional data stream for algorithms that need to manage novelty. For instance, measured reaction time from human subjects can offer insight as to whether a class sample is prone to be confused with a different class -- known or novel. In this work, we designed and performed a large-scale behavioral experiment that collected over 200,000 human reaction time measurements associated with object recognition. The data collected indicated reaction time varies meaningfully across objects at the sample-level. We therefore designed a new psychophysical loss function that enforces consistency with human behavior in deep networks which exhibit variable reaction time for different images. As in biological vision, this approach allows us to achieve good open set recognition performance in regimes with limited labeled training data. Through experiments using data from ImageNet, significant improvement is observed when training Multi-Scale DenseNets with this new formulation: it significantly improved top-1 validation accuracy by 6.02%, top-1 test accuracy on known samples by 9.81%, and top-1 test accuracy on unknown samples by 33.18%. We compared our method to 10 open set recognition methods from the literature, which were all outperformed on multiple metrics.